LLM-as-a-Judge

Some output qualities have no regex. Whether an answer is helpful, on brand, or faithful to its source is a judgment call, and human judgment does not scale to the thousands of outputs a serious eval suite produces. LLM-as-a-judge uses a capable model to grade those outputs, scoring in minutes for cents what would take reviewers days. The research backing is real: in the MT-Bench study, GPT-4's verdicts agreed with human preferences more than 80 percent of the time, the same rate at which the humans agreed with each other. But a judge is itself a fallible model with documented biases, and an uncalibrated judge quietly measures the wrong thing.

Where judges work, and where they break#

Judges are far more reliable at relative judgment than absolute scoring. "Which of these two answers better satisfies the rubric?" plays to a language model's strength: a side-by-side comparison with all the evidence in view. So does rubric grading, where the prompt spells out concrete, checkable criteria. What breaks is absolute scoring without anchors. Ask a judge to rate helpfulness from 1 to 10 and you discover it has no stable internal yardstick: scores cluster in a narrow band, the same answer draws different numbers across runs, and a 7 from one judge model means nothing to another.

A worked example from a support bot. The hopeless version: "score this reply from 1 to 10 for quality." The useful version is a rubric of separate yes/no checks: did it answer the question that was actually asked; is every policy detail consistent with the source document; did it avoid promising anything unauthorized. Each check is a judgment a model can make reliably, and each failure points at a specific fix.

Prefer binary and pairwise over scales#

Two formats cover nearly everything. Pairwise comparison (A or B, which is better for this prompt?) is the right tool for choosing between prompts, models, or pipeline variants. Binary pass/fail per criterion is the right tool for regression gates and production monitoring. Likert scales look more informative but are mostly noise: the gap between a 6 and a 7 is undefined, judges use different bands of the scale inconsistently, and averaging ordinal numbers manufactures false precision.

Hamel Husain's case for binary judgments is also organizational: a forced pass/fail makes the team decide what actually matters in an output. His companion move is to require a short critique alongside every verdict, from both the judge and your human labelers, because those critiques become the few-shot examples that teach the judge prompt what your expert means by "pass."

The biases, and the mitigations#

Judges carry predictable, well-documented biases. Position bias: judges favor an answer based on where it sits in the prompt. Wang and colleagues showed rankings can be flipped simply by swapping the order of two responses; with ChatGPT as judge, reordering let Vicuna-13B beat ChatGPT on 66 of 80 test questions. Mitigation: run every pairwise comparison twice with the order swapped, count a win only when both verdicts agree, and score the rest as ties.

Verbosity bias: judges prefer longer answers even when the extra words add nothing; the MT-Bench authors demonstrated several judge models being fooled by responses padded with repetition. Mitigation: state in the rubric that brevity wins at equal substance, track output length as its own metric, and get suspicious when your "better" variant is also consistently longer.

Self-preference bias (the MT-Bench paper calls it self-enhancement): models rate outputs in their own style more favorably, with the paper observing judges favoring their own generations by roughly 10 to 25 percent higher win rates. Mitigation: never let a model grade its own family's outputs; use a different judge model than the producer, or a small panel of judges from different families. And for any judge, pin the model version and run at temperature zero so scores are at least repeatable.

Calibrate against a human gold set#

An uncalibrated judge is an opinion, not a measurement. The fix is a gold set: 100 to 300 real examples labeled by a domain expert with pass/fail verdicts and short critiques. Run the judge over it and measure judge-human agreement, both raw percent agreement and Cohen's kappa if your labels are skewed toward pass. Then iterate on the judge prompt, folding the expert critiques in as few-shot examples, until agreement stabilizes at a level you trust. For context, human-human agreement in MT-Bench sat around 81 percent, so a judge agreeing with your expert in the low 80s is performing near the ceiling, while one at 60 percent is too close to a coin flip to gate decisions on.

Calibration is not a one-time ceremony. Re-measure whenever the product surface, the judge prompt, or the judge model version changes, and feed every disagreement you find during review back into the gold set, so the reference grows exactly where the judge is weakest.

Designing the pipeline#

A production judge has a recognizable shape. Write a rubric per criterion and ask for one focused judgment per call rather than ten dimensions at once. Require reasoning before the verdict, since producing the evidence first measurably improves reliability (Wang et al. call this multiple evidence calibration, and it mirrors chain-of-thought prompting). Demand structured output, typically JSON with a reasoning field and a pass boolean, so verdicts parse cleanly into dashboards and CI gates. Run pairwise comparisons in both orders automatically. And keep a standing spot-check cadence: a human reviews 20 to 50 judged examples each week, prioritizing verdicts near the decision boundary, with disagreements flowing back into the gold set.

Cost stays manageable because judging is cheap relative to labeling: grading a few thousand outputs with a frontier model costs dollars, and a smaller model that has been validated against your gold set often does the job for cents.

Practical takeaways#

Use judges for what they are good at: pairwise choices and concrete rubric checks, not unanchored scores. Prefer binary verdicts, demand a critique with every verdict, and split fuzzy quality into specific yes/no criteria. Assume position, verbosity, and self-preference bias until your data shows otherwise, and mitigate mechanically: swap orders, watch lengths, and judge with a different model than the one being judged. Above all, calibrate. A judge whose agreement with a human gold set is measured and monitored is an instrument; one without is a vibe.