LLM-as-a-Judge

LLM-as-a-judge means using one language model to grade another model's outputs against a rubric. It exists for the qualities that have no regex: whether an answer is faithful to its source, follows a policy, or strikes the right tone. A person can read a few hundred of those a day, and a serious eval suite produces thousands per run, so a model does the reading and a person checks the model.

We add a judge last. Anything mechanical, such as valid JSON, a required field, or a citation that points at a real passage, gets a code check first, because code does not drift between runs. The judge takes only the questions code cannot answer, and it gates nothing until its agreement with a human expert has been measured. Judge alignment covers that measurement.

Where does LLM-as-a-judge work, and where does it break?#

Judges are far more reliable at relative judgment than at absolute scoring. "Which of these two answers better satisfies the rubric?" gives the model both pieces of evidence side by side. So does a rubric check, where the prompt spells out one concrete criterion. What breaks is a score with no anchors. Ask a judge to rate helpfulness from 1 to 10 and you find it has no stable yardstick: scores cluster in a narrow band, the same answer draws different numbers across runs, and a 7 from one judge model means nothing to another.

Take a support bot. The hopeless version is "score this reply from 1 to 10 for quality". The useful version is three separate yes-or-no checks: did it answer the question that was actually asked, is every policy detail consistent with the source document, and did it avoid promising anything it is not allowed to promise. Each check is a judgment a model can make reliably, and each failure points at a specific fix.

Why binary and pairwise beat a scale#

Two formats cover nearly everything. Pairwise comparison, A or B for this prompt, is the tool for choosing between prompts, models or pipeline variants. Binary pass or fail per criterion is the tool for regression gates and production monitoring. A Likert scale looks more informative and is mostly noise: the gap between a 6 and a 7 is undefined, and averaging the numbers manufactures precision nobody measured.

Hamel Husain's case for binary verdicts is also about the team: a forced pass or fail makes someone decide what matters in an output. He pairs it with a written critique on every verdict, from the judge and from the human labeller, and the expert's critiques become the few-shot examples that teach the judge prompt what "pass" means.

On RepliCoach, our sales-roleplay coach, we went one step further and took the number away from the model. The score comes from a versioned, schema-driven rubric with weighted dimensions, and the model writes only the qualitative read. If the model both writes the feedback and sets the number, a model upgrade shifts every historical comparison. Bigcircle recommends that split for any score you plan to track over months, because the judge model will change before the product does.

Which biases should you assume?#

Assume three until your own data says otherwise, and mitigate each one mechanically.

Position bias means a judge favours an answer for where it sits in the prompt. Wang and colleagues showed that with ChatGPT as the judge, swapping the order of two answers let Vicuna-13B beat ChatGPT on 66 of 80 questions. Run every pairwise comparison twice with the order swapped, count a win only when both verdicts agree, and score the rest as ties.

Verbosity bias means a judge prefers the longer answer when the extra words add nothing. The MT-Bench authors padded 23 answers by restating their own lists, and Claude-v1 and GPT-3.5 fell for it far more often than GPT-4 did. State in the rubric that brevity wins at equal substance, track output length as its own metric, and get suspicious when your "better" variant is also consistently longer.

Self-preference, which the MT-Bench paper calls self-enhancement, means a judge rates its own family's outputs higher. The paper saw GPT-4 favour itself with a 10% higher win rate and Claude-v1 with 25%, though GPT-3.5 did not favour itself. Use a judge from a different family than the model that wrote the answer.

Pin the judge model version too. Temperature zero reduces run-to-run variation without removing it, and some reasoning models do not accept a temperature setting at all, so rerun a fixed set of cases after any judge change and compare.

What does the 80 percent agreement figure mean?#

The number everyone quotes comes from the MT-Bench study: GPT-4's verdicts agreed with human experts over 80% of the time, the same level at which the humans agreed with each other. Read the setup before you borrow it. That figure is 85% for GPT-4 against 81% between humans with ties left out. Counting ties, the rates fall to 66% and 63%. And the task was a general preference between two chatbot answers, not your rubric on your domain.

So the 80% is not a target for your judge. Your ceiling is how often two of your own experts agree on your criterion, and the measure is the judge's true positive rate, true negative rate and Cohen's kappa against one expert's labels, on a hundred to a hundred and fifty cases with failures oversampled. Raw percent agreement on a set where most cases pass says very little, because a judge that passes everything scores high on it.

How should a production judge be built?#

Write one rubric per criterion and ask for one judgment per call rather than ten dimensions at once. Require the reasoning before the verdict: Wang and colleagues found that asking for evidence first made the judge more reliable, which mirrors chain-of-thought prompting. Demand structured output, typically JSON with a reasoning field and a pass boolean, so verdicts parse straight into dashboards and CI gates. Run pairwise comparisons in both orders automatically.

Keep a person in the loop after launch. Bigcircle recommends a standing weekly review of judged cases, weighted toward verdicts near the decision boundary, with every disagreement added to the labelled set, because that set should grow where the judge is weakest. Re-measure agreement whenever the product, the judge prompt or the judge model changes.

The bill rarely decides this. As of October 2026, grading 3,000 outputs at about 2,000 input and 300 output tokens each on Claude Opus 5.5, at $4 and $20 per million tokens, costs roughly $42. The expert's hours spent labelling the cases that validate the judge cost far more, and a smaller judge model that clears the same agreement check costs less again.