PracticeEvaluation & Safety
Judge Alignment (TPR, TNR, Kappa)
Raw agreement hides a judge that passes everything, so alignment needs separate error rates.
A model judge scales evaluation past what a person can read, and it is worth its cost only if it makes the same calls the person would. Measuring that agreement is a step teams skip, and skipping it produces a harness everyone trusts and nobody has checked.
Why raw agreement lies#
Suppose 180 of 200 cases pass, and the judge agrees with the human labels 90% of the time. That sounds like a working judge.
Now consider a judge that returns pass for every input without reading anything. It also scores 90%, because on an imbalanced set the majority class carries the score. This is the most common way a team convinces itself a judge works: the number is high, it is easy to compute, and it stays high no matter how badly the judge handles the cases that matter.
Two error rates, counted separately#
The true positive rate is recall on failures: of the cases the expert failed, how many did the judge catch? The true negative rate is its mirror: of the cases the expert passed, how many did the judge pass?
They are reported apart because they cost different amounts. A false alarm costs an engineer ten minutes of investigation. A missed failure reaches a user. When the two trade against each other, bias toward catching failures.
Cohen's kappa#
Kappa is a single summary that corrects for the agreement you would expect from chance alone. Raw accuracy does not. The always-pass judge scores zero.
| Kappa | Reading |
|---|---|
| below 0.4 | The judge is not measuring your rubric |
| 0.4 to 0.6 | Usable for direction, not for a gate |
| 0.6 to 0.8 | Fine for a gate with a margin |
| above 0.8 | As good as a second labeller |
The human-to-human floor#
Below 0.4, do not start tuning the judge prompt. Have a second person label thirty cases and compute agreement between the two humans first.
If two domain experts only agree at 0.5, no judge will beat that, and what you have found is an underspecified task rather than a broken evaluator. That is a product problem, and prompt engineering does not resolve it.
Practical takeaway#
Validate on a hundred to a hundred and fifty labels with failures deliberately oversampled, because a uniform sample of a hundred on traffic that fails ten percent of the time gives you ten negative examples and a true positive rate too noisy to gate on. Re-run the check whenever the judge model, the rubric or the task changes, since provider updates move judge behaviour silently and nothing in the pipeline announces it.