How to Build LLM Evals

Every guide on this site tells you to build evals first. None of them shows you how. This is the mechanics, and the order matters more than any individual technique in it.

Barnett et al. found the same across research, education and biomedical RAG deployments: "validation of a RAG system is only feasible during operation" (Seven Failure Points, IEEE/ACM CAIN 2024). You cannot certify these systems the way you certify a payments integration. The harness is not a gate you pass once.

What is an eval, exactly?

A test with a fuzzy assertion.

A unit test asserts equality. An eval asserts a property a model has to judge:

# unit test
assert parse_date("2026-09-01") == date(2026, 9, 1)
 
# eval
assert grounded(answer, passages)         # every claim traceable to a passage
assert cites_correctly(answer, passages)  # each citation contains what it claims
assert refused_when_unanswerable(answer, passages)

Everything else is ordinary engineering. It lives in the repo, runs in CI, fails a build, and has a history you can bisect.

Why can't you just write the rubric first?

Because you do not know your criteria until you have looked at outputs, and looking at outputs is what creates them.

Shankar et al. call this criteria drift, and it is the finding the whole method rests on:

To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.

You sit down to write a rubric, you write "the answer should be accurate and helpful", and it is worthless. Then you read forty real outputs and discover the actual criteria were: does it cite the controlling clause rather than a related one, does it flag when two policies conflict, does it refuse when the user's region is not covered. None of those would have occurred to you in advance. All of them are obvious after an hour with the data.

This is why prefab metrics do not work. A generic "helpfulness" or "coherence" score measures an abstraction that has nothing to do with your failure modes. Hamel Husain is blunt about the version of this that shows up most:

If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong.

So the rubric is an output of the process, not an input to it. The process is error analysis.

How do you find your actual failure modes?

Read your traces. Structured, in three passes, borrowed from qualitative research methods.

The first pass is open coding. Pull real traces and write a free-form note on each one that went wrong. No categories yet, and no attempt at consistency. "cited the wrong section", "answered when it should have refused", "used the policy that was superseded in March". You are producing raw observations, not labels.

The second pass is axial coding. Group the notes so similar complaints collapse into a category, and you end with a failure taxonomy of five to ten modes. "cited the wrong section" and "invented a clause number" become one category called citation error, or two, depending on what you saw.

The third pass is counting. How often does each mode occur decides where your engineering time goes. If 43% of failures are retrieval misses, no amount of prompt work matters this quarter.

You stop at theoretical saturation. Husain's practical rule is that if roughly 20 traces turn up no new category you can stop, having reviewed at least 100 to begin with.

The point of this exercise is not the taxonomy. It is that you now know your failure modes empirically rather than from a blog post, and every evaluator you build afterwards targets something you actually observed.

Who decides what good means?

One person. Not a committee.

Husain calls this the principal domain expert: someone with deep domain expertise or who genuinely represents your users. They set the standard. Three stakeholders labelling in parallel produce three rubrics and an argument, and the argument is usually about vocabulary rather than quality.

Consistency matters more than consensus.

The expert's job is small and specific:

1. Read a trace.
2. Make a binary judgement. Pass or fail.
3. Write a critique explaining why.

The critique is not optional and it is where teams cut corners. It has to be detailed enough to be reused as a few-shot example in a judge prompt later. Husain: "being too terse is a common mistake."

The time cost is lower than people fear. Thirty to fifty traces to start, then periodic review. Their involvement is the part that cannot be delegated, because they are the only source of the standard.

How do you write a rubric that holds up?

Once error analysis has given you a taxonomy, each mode becomes a question. Four rules.

One question per judge. A judge asked "is this answer good" is being asked to compound four judgements and will average them into mush. A judge asked "does every factual claim appear in the passages" is close to mechanical.

Ask for a binary decision. Pass or fail, Occasionally three-way when a partial genuinely means something different. Scales fail for a specific reason: nobody can say what separates a 3 from a 4, two labellers will disagree on it, and you need far more samples to detect a real change. Binary decisions force a team to state what actually matters.

Write it from real failures. Every clause in the rubric should trace to something you saw in open coding. If you cannot point at the trace that motivated a clause, delete it.

Put the exclusions in the prompt. Judges drift toward length and fluency, so say so explicitly.

Here is a rubric that came out of the failure note in the table above:

You are checking whether a cited clause governs the question asked.

QUESTION: {question}
PASSAGES: {passages}
ANSWER:   {answer}

For each citation in the ANSWER, decide whether the cited clause is the one
that governs the QUESTION, rather than a related clause on the same topic.

- valid     every citation governs the question asked
- invalid   at least one citation is topically related but not controlling

A correct number quoted from the wrong clause is invalid. Ignore style,
tone, and length entirely.

Return JSON: {"verdict": "...", "offending_citations": ["..."]}

Notice what that rubric could not have been written without: the specific observation that a right number from a wrong clause is the dangerous case, because a reviewer will not catch it. That came from a domain expert's critique, not from a metrics library.

When should you not build a judge?

Most of the time, at first. A model judge is the expensive option and teams reach for it too early.

Start at the top and stop at the first option that catches the failure:

Husain's warning is worth repeating exactly: many teams unnecessarily automate failures they could fix by improving prompts. If a failure mode appears eleven times, fix it. You do not need a permanent instrument to measure a bug you can eliminate this afternoon. Reserve judges for failure modes that persist across fixes.

What should you measure?

Decompose the system before choosing metrics. Retrieval, tool use and generation fail for unrelated reasons and are repaired by unrelated work, and one aggregate score cannot tell you which broke.

Retrieval

If the passage never reaches the model, nothing downstream recovers.

def recall_at_k(retrieved_ids, gold_ids, k):
    return len(set(retrieved_ids[:k]) & set(gold_ids)) > 0
 
def reciprocal_rank(retrieved_ids, gold_ids):
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1 / i
    return 0.0

Measure recall at candidate depth and at final depth. Recall at 50 is what retrieval could deliver; recall at 5 after reranking is what the model sees. The gap is the reranker's contribution and the cheapest place to find points.

Recall says the passage is in the list. MRR says whether it survives context trimming. A system at recall@20 of 0.95 and MRR of 0.2 is retrieving the right thing and burying it.

Anthropic's contextual retrieval numbers give you something to hold your own harness against: 5.7% top-20 failure for standard embedding retrieval, 1.9% with contextual embeddings plus BM25 plus reranking, precision@20 from 0.65 to 0.89.

Tool use

Under-measured, and the place agents fail quietly. Four separate questions:

Sequence is the one people omit and the one that produces agents reporting work as done when a step silently failed.

Generation

Not "is this helpful". Helpfulness is not a measurement, and asking for it produces a judge that agrees with whatever is longest.

Jason Liu, who has consulted on RAG for dozens of startups, makes the ordering argument directly: most teams optimise generation before retrieval works. At recall@k of 0.6, four questions in ten are unanswerable before the model is invoked.

Where do the cases come from?

Error analysis gave you the failure modes. The golden dataset is built to cover them.

Keep it as JSONL so a diff is readable and a case is appendable:

{"id": "refund-001", "query": "What is the refund window for enterprise annual plans?", "gold_passages": ["policy-billing#p14"], "gold_answer": "30 days from invoice date.", "must_refuse": false, "tags": ["billing", "exact-policy"]}
{"id": "refund-002", "query": "Can I get a refund after 90 days?", "gold_passages": ["policy-billing#p14"], "gold_answer": null, "must_refuse": true, "tags": ["billing", "out-of-policy"]}
{"id": "acl-001", "query": "What is the Q3 headcount plan?", "gold_passages": [], "must_refuse": true, "tags": ["permissions"], "as_user": "contractor@example.com"}

Label the passage, not the document

{"gold_passages": ["employee-handbook.pdf"]}                    // useless
{"gold_passages": ["employee-handbook.pdf#p31-parental-leave"]} // testable

A document-level gold label makes retrieval recall look excellent and mean nothing. Any chunk from a 90-page file counts as a hit. Have the expert mark the minimum span that answers the question.

Stratify by the taxonomy you found

Not by traffic volume. Uniform sampling gives you 180 copies of your most common question.

Chroma's generative benchmarking is the systematic way to grow this once the hand-built set exists: golden query-to-chunk pairs measured at token level rather than chunk level.

How do you know your judge is any good?

You measure it against the expert's labels. The number most people reach for is the wrong one.

Say your set is 200 cases and 180 pass. Your judge agrees with human labels 90% of the time. That sounds fine.

Now consider a judge that returns pass for everything without reading. It also scores 90%. Raw agreement cannot separate the two, because the majority class carries the score. Husain says the same thing: "using raw agreement is generally not recommended and can be misleading when classes are imbalanced."

Measure the two error rates separately

The true positive rate is recall on failures: of the cases the expert failed, how many did the judge catch? The true negative rate is the mirror: of the cases the expert passed, how many did the judge pass?

from sklearn.metrics import confusion_matrix, cohen_kappa_score
 
# 1 = expert failed the case (the thing we care about catching)
expert = [1,0,0,1,0,0,0,1,0,0]
judge  = [1,0,0,0,0,0,0,1,0,1]
 
tn, fp, fn, tp = confusion_matrix(expert, judge).ravel()
tpr = tp / (tp + fn)     # 0.67  caught 2 of 3 real failures
tnr = tn / (tn + fp)     # 0.86  passed 6 of 7 good answers
print(cohen_kappa_score(expert, judge))   # 0.51

Those two numbers say different things and cost different amounts:

Low TPR is the expensive failure. A judge at TPR 0.9 and TNR 0.7 is more useful than the reverse, because a false alarm costs an engineer ten minutes and a missed failure reaches a user.

Kappa is still worth computing as a single summary, since it corrects for chance agreement:

How many labels, and when to stop

Start at 30 and continue until you stop learning. Husain: "I start with around 30 examples and keep going until I do not see any new failure modes." For validation, 100 to 150 with failures oversampled. If 10% of traffic fails, a uniform 100 gives you 10 negatives and a TPR you cannot trust.

Three iterations to above 90% agreement is a realistic target, and it is what he reports on a real system.

Below 0.4, check the humans first

Do not tune the judge prompt yet. Have a second person label 30 cases and compute human-to-human agreement. If two experts only agree at 0.5, no judge will beat that. The task is underspecified, and no judge fixes that.

When it fails, split the rubric before touching the model

A judge at 0.45 is usually being asked a compound question. Split it into separate calls:

1. Does every factual claim appear in the passages?      grounded | not_grounded
2. Does each citation govern the question asked?         valid | invalid
3. Given the passages, should this have refused?         should_answer | should_refuse

Compound them in code afterwards. This is usually the largest single gain available and needs no prompt tuning beyond the split.

Three biases to expect

Re-run alignment whenever you change the judge model, the rubric, or the task. Provider updates move judges silently and nothing in your pipeline announces it.

How do you stop overfitting the set?

You will overfit it. The set is small, you read the failures, you fix exactly those.

When dev keeps improving and test does not move, you are tuning to the cases rather than the task. You cannot see that divergence without the split.

Then rotate. Retire cases the system has passed for three months and replace them with fresh production failures. A set that never changes measures a system that stopped changing.

What does this look like in CI?

# .github/workflows/evals.yml
on:
  pull_request:
    paths: ["prompts/**", "retrieval/**", "chunking/**", "tools/**", "evals/**"]
 
jobs:
  evals:
    steps:
      - run: uv run evals --set evals/dev.jsonl --report out/report.json
      - run: uv run evals-gate out/report.json
        # fails if:
        #   recall@20        < 0.90
        #   groundedness_tpr < 0.90
        #   refusal_recall   < 0.98   (missed refusals are the expensive failure)
        #   p95_latency      > 2500ms

Watch the path filter. Chunking and retrieval config change quality as much as the prompt, and a filter that fires only on prompts/** lets the expensive changes through ungated.

Version the set in the repo. It is source, so it gets reviewed and bisected like source.

Write per-case results rather than only a score. You need to diff two runs and see which five cases flipped. An aggregate moving 0.91 to 0.89 tells you nothing about where to look.

Pin everything. Temperature 0, pinned model versions for the system and the judge, versions recorded in the report.

On our DevOps agent this loop took template errors before deploy from 10% to 4%. Not a better model. A closed validate-and-apply loop with every failure fed back into the case set.

On the trademark platform the harness is what keeps a fine-tuned model changeable. Without it nobody would touch a model an attorney has signed off, and the system would be frozen at launch quality.

The loop this all becomes

Error analysis is not a phase you complete. Production traces produce new failure notes, notes cluster into modes, modes become cases, cases gate deployments, deployments produce traces.

Liu's framing is the one to manage against: quality is a lagging metric, experiments per week is the leading one. A team running four experiments a week will pass a team running one, whatever either believes about its approach.

What will evals not tell you?

How do we run this, in order?

1. Collect 100 real traces. Production if you have it, whoever asked for the feature if you do not.
2. Open coding. Free-form note on each failure. No categories.
3. Axial coding. Cluster into five to ten modes. Count them.
4. Stop at saturation. 20 fresh traces, no new category.
5. Name a principal domain expert. One person. Binary judgements with written critiques.
6. Fix what is cheap. Eleven instances of one bug is a fix, not an instrument.
7. Write assertions and reference checks for everything mechanical.
8. Build a judge only for the modes that persist, one question each, binary, exclusions in the prompt.
9. Validate it. 100 labels, oversample failures, report TPR and TNR. Below κ 0.4, split the rubric.
10. Split dev and test 70/30. Gate CI on the wide path filter.
11. Feed every production failure back in.

Steps 1 to 4 are a day and they determine whether the rest is worth building.

Sources and further reading

• Hamel Husain, Creating a LLM-as-a-Judge That Drives Business Results and the LLM Evals FAQ. The principal domain expert, binary judgements with critiques, and judge validation.
• Shankar et al. on criteria drift, via the same FAQ. The finding that grading defines the criteria.
• Barnett et al., Seven Failure Points When Engineering a RAG System, IEEE/ACM CAIN 2024.
• Eugene Yan, Task-Specific LLM Evals that Do and Don't Work.
• Jason Liu, RAG series. Retrieval before generation, experiments per week.
• Anthropic, Contextual Retrieval.
• Chroma, Evaluating Chunking Strategies and Context Rot.
• Ragas metrics for standard definitions.
• Ours: why RAG systems fail in production and why AI agents report unfinished work as done.

Production takeaways

• You cannot write the rubric first. Criteria emerge from grading, and grading is what defines them.
• Open coding, then axial coding, then counts. Build the evaluator for the mode that dominates.
• Stop at saturation: 20 traces with no new category, after at least 100 reviewed.
• One principal domain expert. Consistency beats consensus.
• Binary judgements with critiques detailed enough to become few-shot examples.
• Assertions and reference checks before judges. A bug that appears eleven times is a fix, not an instrument.
• Report TPR and TNR separately. A missed failure costs more than a false alarm.
• Below κ 0.4, check human-to-human agreement before blaming the judge.
• Split dev and test. When dev improves and test does not, you are tuning to the cases.

Where to start tomorrow

Pull a hundred traces and read them with a text file open beside you.

Write a sentence about every one that went wrong. Do not categorise yet.