PracticeEvaluation & Safety
Judge Alignment (TPR, TNR, Kappa)
At a glance
Raw agreement hides a judge that passes everything, so alignment needs separate error rates.
- Who this is for
- Engineers and technical readers learning the terms used in AI systems.
- Topics
- Evaluation & Safety
- Practice
In short
Judge alignment is whether a model judge makes the same calls as the expert it stands in for, measured with separate error rates and an agreement score corrected for chance. Bigcircle will not gate on raw agreement, oversamples failures in the label set, and reruns the check when the judge model, the rubric, or the task changes. Our guide on how to build LLM evals reports the two error rates separately, because raw agreement looks strong when most cases pass.
Key facts
Definition
Judge alignment is the measured match between a model judge and a human expert, reported as true positive rate, true negative rate, and Cohen's kappa so an always-pass judge cannot look competent.
How Bigcircle does it
- Bigcircle labels a hundred to a hundred and fifty cases with failures deliberately oversampled, because a uniform draw from traffic that fails ten percent of the time leaves too few failures to trust the true positive rate.
- Report how many expert failures the judge caught and how many expert passes it also passed, and prefer catching failures when the two trade off, because a miss reaches a user and a false alarm costs an engineer ten minutes.
- Below a kappa of 0.4, have a second person label thirty cases before editing the judge prompt, because disagreement between experts means the task is underspecified.
Comparison
Raw percent agreement stays high on a set where most cases pass, even for a judge that never reads the input, while Cohen's kappa subtracts chance agreement and scores that always-pass judge at zero.
Key number
Worked example on the page: if 180 of 200 cases pass, a judge that agrees with the labels 90% of the time can be an always-pass judge, which also scores 90% because the majority class carries the number.
A model judge scales evaluation past what a person can read, and it is worth its cost only if it makes the same calls the person would. Measuring that agreement is a step teams skip, and skipping it produces a harness everyone trusts and nobody has checked.
Why raw agreement lies#
Suppose 180 of 200 cases pass, and the judge agrees with the human labels 90% of the time. That sounds like a working judge.
Now consider a judge that returns pass for every input without reading anything. It also scores 90%, because on an imbalanced set the majority class carries the score. This is the most common way a team convinces itself a judge works: the number is high, it is easy to compute, and it stays high no matter how badly the judge handles the cases that matter.
Two error rates, counted separately#
The true positive rate is recall on failures: of the cases the expert failed, how many did the judge catch? The true negative rate is its mirror: of the cases the expert passed, how many did the judge pass?
They are reported apart because they cost different amounts. A false alarm costs an engineer ten minutes of investigation. A missed failure reaches a user. When the two trade against each other, bias toward catching failures.
Cohen's kappa#
Kappa is a single summary that corrects for the agreement you would expect from chance alone. Raw accuracy does not. The always-pass judge scores zero.
| Kappa | Reading |
|---|---|
| below 0.4 | The judge is not measuring your rubric |
| 0.4 to 0.6 | Usable for direction, not for a gate |
| 0.6 to 0.8 | Fine for a gate with a margin |
| above 0.8 | As good as a second labeller |
The human-to-human floor#
Below 0.4, do not start tuning the judge prompt. Have a second person label thirty cases and compute agreement between the two humans first.
If two domain experts only agree at 0.5, no judge will beat that, and what you have found is an underspecified task rather than a broken evaluator. That is a product problem, and prompt engineering does not resolve it.
Practical takeaway#
Validate on a hundred to a hundred and fifty labels with failures deliberately oversampled, because a uniform sample of a hundred on traffic that fails ten percent of the time gives you ten negative examples and a true positive rate too noisy to gate on. Re-run the check whenever the judge model, the rubric or the task changes, since provider updates move judge behaviour silently and nothing in the pipeline announces it.