In LLM-as-a-judge, a team gives a language model a rubric, the original request, and one or more responses, then asks it to score the response or pick the better of two. The method scales evaluation for open-ended outputs, such as summaries or support replies, where exact-match metrics fail and human review is slow. It is now a common part of LLM evaluation and agent evaluation pipelines.
Judge models have known biases. Research has documented a preference for the first answer shown, for longer answers, and for answers written in a model’s own style. Teams reduce these effects by swapping answer order, writing specific rubrics, and checking judge scores against a sample of human labels. A judge also scores what it is shown. A well-written answer built on outdated or wrong records can still receive a high score, so the records behind each answer need their own check.