How to build LLM evals

Build the eval from failures you have actually read, rather than from a rubric you wrote in advance. Group those failures, put one domain expert in charge of the standard, and create the cases only after that. Use code where the property is mechanical, and add a model judge only after it agrees with that expert. Bigcircle uses this order on client systems, because a score you cannot trace to a failure does not tell you what to fix.

Every guide on this site tells you to build evals before you do anything else, and none of them has explained how. That gap is worth closing, because the order of the steps matters more than any individual technique in them, and most of the teams we work with have the order backwards.

The difficulty is not that evaluation is conceptually hard. It is that the thing you are trying to measure does not hold still. A traditional test suite asserts that a function returns four, and four is four regardless of who is asking. An eval asserts that an answer is grounded, or that a citation supports the claim attached to it, or that a system correctly declined to answer a question it had no evidence for. Those are judgements, they vary between reasonable people, and they change as you learn more about your own product. Barnett et al. reached the same conclusion across research, education and biomedical deployments: "validation of a RAG system is only feasible during operation" (Seven Failure Points, IEEE/ACM CAIN 2024). You cannot certify these systems the way you would certify a payments integration, which means the harness is not a gate you pass once but an instrument you keep and maintain.

In this guide we walk through the process we use on client systems, in the order we use it. It starts with reading real failures before writing any criteria, grouping those failures into a taxonomy, and deciding who owns the standard. From there you build the case set, choose what to measure on it, and decide which of those checks need a model at all. Then you write the rubric for the ones that do, and validate that judge against human labels before it gates anything. We use numbers from two of our own production systems where they illustrate the point.

What is an eval, exactly?

An eval is a test whose assertion requires judgement rather than equality.

The distinction matters more than it first appears. A unit test can be written before the feature exists, because the correct answer is knowable in advance. An eval usually cannot, because the definition of a correct answer is something you discover by looking at wrong ones. Everything else about an eval is ordinary engineering: it lives in the repository, it runs in continuous integration, it fails a build, and it has a history you can bisect when a regression appears.

# unit test
assert parse_date("2026-09-01") == date(2026, 9, 1)
 
# eval
assert grounded(answer, passages)         # every claim traceable to a passage
assert cites_correctly(answer, passages)  # each citation contains what it claims
assert refused_when_unanswerable(answer, passages)

Why can't you just write the rubric first?

Because you do not know your criteria until you have looked at outputs, and looking creates them.

This is not a failure of discipline, and a more rigorous team does not avoid it. Shankar et al. identified it as a structural property of the task and named it criteria drift:

To grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria.

We see this on nearly every engagement. A team sits down to write a rubric before launch and produces something like "the answer should be accurate and helpful." An hour of reading real outputs turns that into something far more specific. Does the response cite the controlling clause rather than a topically related one? Does it flag a conflict between two policies instead of silently picking one? Does it decline when the user's jurisdiction is not covered by the documents in scope? None of those would have occurred to anyone in advance. All of them are obvious in retrospect.

This is also why prefabricated metrics tend not to survive contact with a real product. A generic helpfulness or coherence score measures an abstraction that has no particular relationship to the failure modes your system exhibits, and when it moves you cannot say which change moved it. We have watched a team spend a sprint chasing a coherence score that fell after a retrieval change, on a system whose actual problem was that the reranker had started dropping the controlling clause. The score was real. It just was not measuring the thing that broke.

The rubric, then, is an output of the process rather than an input to it. The process that produces it is error analysis.

How do you find your actual failure modes?

By reading your traces in a structured way, in three passes borrowed from qualitative research methods. The structure exists because unstructured reading produces impressions rather than counts, and impressions cannot tell you where to spend a quarter.

The first pass is open coding, in which you pull real traces and write a free-form note on each one that went wrong. There are no categories at this stage and no attempt at consistency, because imposing either would defeat the purpose. Notes look like "cited the wrong section," "answered when it should have refused," or "used the policy that was superseded in March." What you are producing is raw observation, and the discipline is to resist labelling while you are still looking.

The second pass is axial coding, where you group those notes so that similar complaints collapse into a category and you end with a failure taxonomy of roughly five to ten modes. Whether "cited the wrong section" and "invented a clause number" become one category called citation error or two separate ones depends entirely on what you saw, and that decision is the substantive work of the pass.

The third pass is counting, and it decides where engineering time goes. If forty-three percent of your failures are retrieval misses, no amount of prompt work will move your quality this quarter. Counting makes that unarguable rather than a matter of opinion between two engineers who each read a handful of examples.

You stop at theoretical saturation, which in practice means twenty consecutive traces producing no new category, with at least a hundred reviewed before you trust that. Stopping earlier is the common error, and it produces a taxonomy that describes the sample rather than the system. The modes you found are real, but the ones you missed are the ones that will show up in production. The point of the exercise is not the taxonomy document. It is that you now know your failure modes empirically rather than from a blog post, and every evaluator you build afterwards targets something you have actually observed in your own system.

Who decides what good means?

One person, and not a committee.

Pick one principal domain expert: someone with real depth in the domain, or who genuinely represents the users the system is built for. Their job is to set the standard, and it has to be one person because three stakeholders labelling in parallel produce three rubrics and an argument. We have sat through that argument more than once, and it is almost always about vocabulary rather than about quality, which means it consumes a week and settles nothing. Consistency matters here more than consensus does.

The work itself is small and specific. The expert reads a trace, makes a binary pass-or-fail judgement, and writes a critique explaining the reasoning behind it. Teams cut the critique when they are busy, and cutting it costs more than it saves, because those critiques become the few-shot examples in the judge prompt later. On the trademark platform the useful ones named the clause that should have governed and said why the wrong one was plausible, which is the reasoning a judge needs in order to make the same call. The difference between a usable critique and a useless one is large:

The time cost is lower than teams tend to fear. Thirty to fifty traces to establish the standard, then periodic review as the system changes. What cannot be delegated is the judgement itself, because the expert is the only available source of the standard the system is being held to.

Where do the cases come from?

You have a taxonomy and someone who owns the standard. The next thing to build is the set of cases both will be applied to, and it is built to cover the modes the taxonomy named rather than to sample traffic. Keep it as JSONL, so a diff is readable and a new case is one appended line.

{"id": "refund-001", "query": "What is the refund window for enterprise annual plans?", "gold_passages": ["policy-billing#p14"], "gold_answer": "30 days from invoice date.", "must_refuse": false, "tags": ["billing"]}
{"id": "refund-002", "query": "Can I get a refund after 90 days?", "gold_passages": ["policy-billing#p14"], "gold_answer": null, "must_refuse": true, "tags": ["out-of-policy"]}
{"id": "acl-001", "query": "What is the Q3 headcount plan?", "gold_passages": [], "must_refuse": true, "tags": ["permissions"], "as_user": "contractor@example.com"}

Two decisions in that format do most of the work.

Label the passage, not the document. This is the mistake most worth avoiding early:

{"gold_passages": ["employee-handbook.pdf"]}                    // useless
{"gold_passages": ["employee-handbook.pdf#p31-parental-leave"]} // testable

A document-level gold label makes retrieval recall look excellent while measuring nothing, because any chunk drawn from a ninety-page file counts as a hit. Ask the expert to mark the minimum span that answers the question, and if two spans are required, record both, which turns the case into a multi-hop test at no extra cost.

Stratify the set by the taxonomy you found, not by traffic volume. A uniform sample of production traffic returns whatever your users happen to ask most often, which is usually one easy intent repeated many times.

Chroma's generative benchmarking work is the systematic way to grow this once a hand-built set exists, pairing golden queries with the chunks that should be retrieved and measuring at token level rather than chunk level.

What should you measure?

With cases in hand, you can decide what to measure on them. Decompose the system first. Retrieval, tool use and generation fail for unrelated reasons and are repaired by unrelated work, so a single quality score cannot tell you which of the three broke on any given change.

Here is the whole set in one place, before we go through them. Standard names are given where one exists, because you will meet these under those names in Ragas and elsewhere.

Two notes on that table. The gates are the ones we start from and then move once a system has its own history; they are not standards. And precision does not appear on the retrieval rows, because at the depths that matter here recall is the constraint. Precision starts to matter when you are paying per passage in a long context window.

Retrieval#

If the passage never reaches the model, nothing downstream can recover, which is why this layer is measured first and why a low number here makes every other measurement uninterpretable.

def recall_at_k(retrieved_ids, gold_ids, k):
    return len(set(retrieved_ids[:k]) & set(gold_ids)) > 0
 
def reciprocal_rank(retrieved_ids, gold_ids):
    for i, doc_id in enumerate(retrieved_ids, start=1):
        if doc_id in gold_ids:
            return 1 / i
    return 0.0

Measure recall at your candidate depth and again at your final depth, because the two numbers answer different questions. Recall at fifty tells you what retrieval could deliver if nothing downstream discarded it, and recall at five after reranking tells you what the model actually receives. The gap between them is the reranker's contribution, and in our experience it is the cheapest place in the whole pipeline to find points.

Recall tells you the passage is somewhere in the list, while MRR tells you whether it will survive context trimming. A system reporting recall at twenty of 0.95 alongside an MRR of 0.2 is retrieving the right thing and then burying it, and the two numbers together diagnose that in a way neither does alone.

Anthropic's contextual retrieval benchmarks give you a target to hold your own harness against. Standard embedding retrieval misses 5.7% of top-twenty retrievals, contextual embeddings plus BM25 plus reranking takes that to 1.9%, and precision at twenty rises from 0.65 to 0.89.

Measure this before you touch generation quality, because the order is not a preference. At a recall of 0.6, four questions in ten are unanswerable before the model is ever invoked, so every prompt change you evaluate after that point is being measured against a ceiling you have not lifted. On Prospex the retrieval pipeline reduced 420 candidate sources to the 6 that answered the question, and that reduction made the downstream answer quality worth measuring at all. Once those numbers are readable, which layer to change is the decision in RAG vs fine-tuning vs agentic search.

Tool use#

This layer is consistently under-measured, and it is where agents fail quietly rather than loudly. Four separate questions need separate answers.

Sequence is the one teams omit, and it produces agents reporting work as done when a step in the middle failed silently.

Safety: guardrails and PII#

A guardrail and an eval are different mechanisms and teams keep conflating them. A guardrail runs on every request and blocks or rewrites. An eval runs against a fixed set and tells you whether the guardrail works. You do not pick one. You build the guardrail and then measure it, and teams skip the measuring.

Three kinds of case go in the set, and the third is the one teams miss.

Cases the guardrail should catch. A request that would send personal data to a provider, an action outside the policy, an instruction buried in a retrieved document. These are the obvious ones.

Cases it should let through. Over-blocking is a real failure with a real cost, and a guardrail tuned only on the first kind of case will get quieter and quieter until the product stops working. If nobody is measuring the allowed traffic, nobody notices.

Cases where the sensitive data arrives from retrieval rather than from the user. A redaction layer that reads the request and not the retrieved passages is a layer with a hole in it, and the hole only opens once the corpus grows.

PII handling has a second measurement most sets miss. If the boundary encodes on the way out and decodes on the way back, the eval has to check both directions. Redaction that never decodes cleanly produces answers that are safe and useless, and that failure shows up as a quality complaint rather than a safety one, so it gets routed to the wrong team.

That last row makes the rest possible, because a guardrail that logs only its blocks cannot be measured: the cases you most need are the ones it let through and should not have.

The gates here are not like the others in this guide. Elsewhere we start at 0.90 and move it once a system has history. On safety the target is 1.0 and a miss is an incident, because the cost of one leaked record is not on the same scale as one wrong answer.

Generation#

Notice that helpfulness does not appear, because it is not a measurement, and a judge asked to assess it will reliably agree with whichever answer is longest.

One reframing worth adopting alongside these, which we take from Jason Liu: treat quality as the lagging metric and experiments per week as the leading one. You report quality to stakeholders, but experiment rate is the one a team can manage directly, and it predicts where quality lands next quarter. A team running four experiments a week will overtake a team running one regardless of what either believes about its own approach.

When should you not build a judge?

You now know what to measure, so next comes what does the measuring, and for most of these checks it should not be a model. A judge is the expensive option, to build and then to run on every case forever, and teams reach for one earlier than the evidence justifies.

The mistake we see most often is a team building a judge for a failure they could have fixed that afternoon. If a mode appears eleven times in your traces and a prompt change removes it, you have spent nothing and gained the fix. Build a judge for it instead and you have added a per-case cost that runs for as long as the product does, all to keep measuring something that stopped happening. Judges earn their cost on the modes that survive several attempted fixes.

How do you write a rubric that holds up?

For the modes that survive, you need a judge, and a judge is only as good as the question you give it. Each mode becomes one question the rubric asks. Four things decide whether that rubric works.

Two of those are worth more than a table row.

Ask one question at a time. "Does every claim appear in the passages?" fails in a way you can act on, because you go and check the claims. "Is this answer good?" gives you a number that dropped and no way to tell whether retrieval, the citation, or a missing refusal caused it.

Binary is harder to agree on than it looks, and that is the point. Two people will argue about where the line sits, and the argument is worth having. A team that cannot say whether an answer passes does not yet know what it wants from the system.

A three-way split is sometimes right, when a partial answer means something real. A scale is not. Ask two people to score the same answer and one says three and the other says four. A real change has to rise above that noise before anyone can see it, and that takes more labels than most teams will ever collect.

Here is a rubric that came directly out of the failure note in the table above:

You are checking whether a cited clause governs the question asked.

QUESTION: {question}
PASSAGES: {passages}
ANSWER:   {answer}

For each citation in the ANSWER, decide whether the cited clause is the one
that governs the QUESTION, rather than a related clause on the same topic.

- valid     every citation governs the question asked
- invalid   at least one citation is topically related but not controlling

A correct number quoted from the wrong clause is invalid. Ignore style,
tone, and length entirely.

Return JSON: {"verdict": "...", "offending_citations": ["..."]}

You could not write that rubric without one thing someone noticed first: a right number pulled from the wrong clause is the case that hurts, because a reviewer skimming the output will not catch it. That came from a domain expert's critique. No off-the-shelf metric would have found it.

How do you know your judge is any good?

You measure it against the expert's labels, and the number most teams reach for first is the wrong one.

Suppose your set contains two hundred cases of which a hundred and eighty pass, and your judge agrees with the human labels ninety percent of the time. That sounds like a working judge. Now consider a judge that returns pass for every input without reading anything at all, which also scores ninety percent on that set. Raw agreement cannot distinguish between the two, because on an imbalanced set the majority class carries the score entirely. This is the single most common way a team convinces itself a judge works. The number is high, it is easy to compute, and it stays high no matter how badly the judge handles the cases that matter.

Measure the two error rates separately#

The true positive rate is recall on failures, meaning of the cases the expert failed, how many the judge caught. The true negative rate is its mirror, meaning of the cases the expert passed, how many the judge also passed. Report them separately and the judge's behaviour becomes legible, because the two failures cost very different amounts.

from sklearn.metrics import confusion_matrix, cohen_kappa_score
 
# 1 = expert failed the case (the thing we care about catching)
expert = [1,0,0,1,0,0,0,1,0,0]
judge  = [1,0,0,0,0,0,0,1,0,1]
 
tn, fp, fn, tp = confusion_matrix(expert, judge).ravel()
tpr = tp / (tp + fn)     # 0.67  caught 2 of 3 real failures
tnr = tn / (tn + fp)     # 0.86  passed 6 of 7 good answers
print(cohen_kappa_score(expert, judge))   # 0.51

A low true positive rate is the expensive failure. A judge at 0.9 TPR and 0.7 TNR costs an engineer ten minutes on a false alarm, while the reverse lets a real defect reach a user. When the two trade against each other, we bias toward catching failures.

Kappa is still worth computing as a single summary, because it corrects for chance agreement in a way raw accuracy does not.

How many labels, and when to stop#

Start at thirty and keep going until you stop learning. For validating a judge specifically, a hundred to a hundred and fifty labels with failures deliberately oversampled is the working range, and the oversampling is not optional. If ten percent of your traffic fails, a uniform sample of a hundred gives you ten negative examples. That puts an interval around your true positive rate wide enough that the number cannot support a release decision. Expect three iterations of rubric and prompt before agreement is high enough to gate on.

Below 0.4, check the humans before the judge#

Do not begin tuning the judge prompt. Have a second person label thirty cases and compute human-to-human agreement first. If two domain experts only agree at 0.5, no judge will beat that, and what you have found is an underspecified task rather than a broken evaluator. That is a product problem, and no amount of prompt engineering resolves it.

When it fails, split the rubric before touching the model#

A judge scoring around 0.45 is usually being asked a compound question, and the fix is to decompose it into separate calls that each ask one thing:

1. Does every factual claim appear in the passages?      grounded | not_grounded
2. Does each citation govern the question asked?         valid | invalid
3. Given the passages, should this have refused?         should_answer | should_refuse

Each of those is close to mechanical, and the results are compounded afterwards in code. This is usually the largest single improvement available, and it requires no prompt tuning beyond the split itself.

Three biases to expect#

Judges prefer longer answers, so if a change made your outputs longer, some of the apparent improvement is an artefact; bucket by response length and check that the score holds within each bucket. In pairwise comparison the first candidate wins more often than it should, so run both orders and average. And a judge prefers text from its own model family, so if one provider both writes and judges, score a holdout sample with a different provider and look for disagreement concentrated on your own model's output.

Re-run the alignment check whenever you change the judge model, the rubric, or the task itself. Provider updates move judge behaviour silently, and nothing in your pipeline will announce that it has happened.

How do you stop overfitting the set?

You will overfit it, because the set is small, you read the failures, and you fix precisely those. Splitting it is the standard defence and it works here for the same reason it works anywhere else.

When the dev split keeps improving and the test split does not move, you are tuning to the cases rather than to the task, and that divergence is invisible without the split. Beyond that, retire cases the system has passed for three months and replace them with fresh production failures, because a set that never changes is measuring a system that stopped changing.

Is the difference real?

Usually not, at the set sizes teams actually have. Eval sets are small and small sets are noisy, and a hundred cases scoring 0.90 carry a confidence interval of about six points either side. A run at 0.91 and a run at 0.89 are therefore the same result as far as the evidence goes, and teams ship on that difference every week.

Comparing the two runs as separate samples makes the problem worse than it has to be. Detecting a move from 0.89 to 0.91 that way needs somewhere near three and a half thousand cases, which no team building a product is going to label.

Both runs went over the same cases, though, so they were never separate samples to begin with. Throw away every case that scored the same in both and count only the ones that changed verdict.

Six cases moving the same way is enough at the usual five percent threshold, because a change that did nothing would have split them roughly evenly. That is also why ten better against two worse is weaker evidence than eight against one. This is a sign test and any statistics library will run it. It is also the second reason to write per-case results instead of an aggregate, since the aggregate is the one number that cannot tell you whether the move was real.

Two limits are worth holding on to. The test tells you the direction of the change is real, and says nothing about whether the size of it is worth shipping. And thirty cases can only produce six flips if the change touched a fifth of them, so a small set still hides small improvements. If you need to see a two-point move you need a few thousand cases, and the better question is why a two-point move is a release decision at all.

What does this look like in CI?

Deliberately ordinary, which is the point: a harness nobody runs is worth nothing, so the eval suite goes where every other check goes and fails a build the same way.

# .github/workflows/evals.yml
on:
  pull_request:
    paths: ["prompts/**", "retrieval/**", "chunking/**", "tools/**", "evals/**"]
 
jobs:
  evals:
    steps:
      - run: uv run evals --set evals/dev.jsonl --report out/report.json
      - run: uv run evals-gate out/report.json
        # fails if:
        #   recall@20        < 0.90
        #   groundedness_tpr < 0.90
        #   refusal_recall   < 0.98   (missed refusals are the expensive failure)
        #   p95_latency      > 2500ms

The path filter is the part worth paying attention to, because chunking configuration and retrieval settings change output quality as much as the prompt does. A filter that only fires on prompts/** will let the most expensive changes through ungated.

Three practices matter more than the framework you choose. Version the eval set in the repository, because it is source: it gets reviewed, and you can bisect it. Write per-case results rather than only an aggregate, because you need to diff two runs and see which five cases flipped, and a score moving from 0.91 to 0.89 tells you nothing about where to look. And pin everything, meaning temperature at zero, pinned model versions for both the system and the judge, and those versions recorded in the report, since otherwise you cannot attribute a regression to your own change.

Temperature zero plus pinned versions reduces variation and does not remove it, so a case that has to come back the same still needs the repeat-run check in why temperature zero doesn't make an LLM agent repeatable.

On our DevOps agent this loop took template errors before deploy from ten percent to four percent. The model did not change; what changed was a closed validate-and-apply loop with every failure fed back into the case set. On the trademark platform the harness keeps a fine-tuned model changeable at all. Without it nobody would touch a model an attorney has signed off on, and the system would sit frozen at whatever quality it launched with.

Where does online evaluation fit?

Online evaluation tells you whether your offline score means anything. Everything up to here is offline evaluation. You wrote the cases, an expert wrote down the right answers, and CI scores one against the other on every pull request. The other half is online evaluation, which watches the shipped system answer questions nobody wrote an answer key for.

Most teams build the first column and stop there, which is where this guide has left you, and it sits awkwardly against the claim the guide opened with: validation of these systems is only feasible during operation.

Your offline score is a proxy for how the system behaves in front of users, and a proxy needs checking. The check is short to state: when the eval improves and nothing moves in production, your cases do not represent your traffic. That is a finding about the set rather than about the model, and the response is to go back to error analysis with fresh traces instead of tuning the prompt again.

So you need a production signal to compare against, and ratings will not give you one. Almost nobody clicks the thumb, and the ones who do are the angriest and the most delighted. A study of the WildChat and LMSYS conversation logs found this feedback works as a lens on your users rather than as a clean training target (User Feedback in Human-LLM Dialogues, EMNLP 2025).

Behaviour arrives at volume instead: did the user ask the same question again in different words, abandon the session, hit retry, edit the answer before sending it, or escalate to a human? Any one of them is ambiguous, since someone copying an answer might be satisfied or might be collecting evidence for a complaint. Read against a baseline and in aggregate they move when quality moves, which is the only property you need from them.

Those signals also decide which traces you read. You get thousands a week and can read a hundred, so something has to choose which hundred. Take the traces carrying a negative signal, the outliers with unusual retry counts or session lengths, and the ones your judge scored badly. Then keep a slice of plain random traces in every batch, because the random slice is the only part that can find a failure mode your current signals do not describe. Teams sustain this at ten to twenty traces a week between rounds, plus a hundred fresh ones each time they run full error analysis.

On the DevOps agent this was easier than it usually is. The production signal was mechanical: a generated template either applied to AWS or it did not, so every failure wrote itself into the case set without anyone deciding it was one. Most products do not get that, so pick two behavioural signals and instrument them before you need them. Treat the gap between those and your eval score as the thing you are managing.

The loop this all becomes

Error analysis is not a phase you complete and move past. Production traces produce new failure notes, those notes cluster into modes, the modes become cases, the cases gate deployments, and the deployments produce more traces. The teams that sustain quality over years are the ones that closed this loop early, rather than the ones that built the most sophisticated evaluator.

What will evals not tell you?

A correct transcript can still be a failed call, so a voice product needs checks on the audio beside this text harness. Turn-taking, interruptions, intelligibility, and silence are scored on their own, and voice AI evals keeps those verdicts off the text score. Treat that set as its own cases rather than another row in the table above.

A fan-out needs the same kind of separate check. Score the merged result against the expected state, which when to fan out to many agents sets out, rather than the path each worker happened to take.

How do we run this, in order?

1. Collect a hundred real traces, from production if you have it and from whoever asked for the feature if you do not.
2. Open code them, writing a free-form note on each failure with no categories.
3. Axial code, clustering into five to ten modes, then count them.
4. Stop at saturation, meaning twenty fresh traces produce no new category.
5. Name a principal domain expert, one person, making binary judgements with written critiques.
6. Fix what is cheap. Eleven instances of one bug is a fix, not an instrument.
7. Write assertions and reference checks for everything mechanical.
8. Build a judge only for the modes that persist, one question each, binary, with the exclusions in the prompt.
9. Validate it against a hundred labels with failures oversampled, reporting TPR and TNR. Below κ 0.4, split the rubric.
10. Split dev and test seventy-thirty, and gate CI on the wide path filter.
11. Compare runs by counting flipped cases, not by reading the aggregate.
12. Instrument two behavioural signals in the product, and check the eval moves with them.
13. Feed every production failure back into the set.

Steps one through four take a day, and they determine whether everything after them is worth building.

Production takeaways

• Label the span rather than the document, because a document-level gold label passes everything.
• Stratify the set by failure mode, since a uniform sample is a hundred and eighty copies of one question.
• Measure recall at k before anything else. At 0.6, four questions in ten are already lost.
• Rubrics ask for a decision between three options, never a rating out of ten.
• Raw agreement lies on imbalanced sets. Report TPR and TNR, and read the confusion matrix.
• Below κ 0.4, check human-to-human agreement before blaming the judge.
• Split dev and test. When dev improves and test does not, you are tuning to the cases.
• Gate CI on chunking and retrieval paths, not only on prompts.
• Count the cases that flipped. On a hundred cases, 0.91 against 0.89 is noise.
• Offline evals gate a release. Online signals decide which traces you read next.
• An offline win that moves nothing in production means your cases are wrong, not the model.

Where to start tomorrow

Open your logs and copy thirty real queries into evals/dev.jsonl.

Then ask the person who knows the domain to mark the paragraph that answers each one.

Common questions

How do I build evals for an LLM app?#

Start from real traces and write a note on each failure before you name a category. Cluster the notes and have one expert mark pass or fail, with a critique, then build cases for those modes. Use code for anything mechanical, and let a model judge block a release only after it agrees with the expert.

What is an LLM eval?#

An eval is a test whose pass or fail takes a judgement rather than an equality check. Grounded, correctly cited, and correctly refused are judgements, and two careful readers can disagree until the line is written down. The harness still lives in the repository, runs in continuous integration, and leaves a history you can bisect when a score moves.

Why can't I write the rubric first?#

You learn the criteria by reading outputs, and that reading makes them specific. A rubric written before launch tends to ask for an answer that is accurate and helpful, which does not tell you what to change. An hour with real traces turns that into a concrete question, such as whether the response cited the clause that governs.

How many traces should I read?#

Review at least a hundred traces, and stop when twenty fresh ones add no new failure category. Stopping earlier describes the sample you happened to open, and the modes you missed are the ones that show up later in production. Count after the categories exist, since a pile of retrieval misses will not move if the quarter goes to the prompt.

When should I not use an LLM as a judge?#

Use code when the check is mechanical, such as valid structure, a citation that exists, or a forbidden string. Use a reference check when you already have a gold answer or a gold set of passages. A model judge takes days to build and a cost on every later case, so reserve it for modes that survive several fixes.

How do I know the judge is good enough to gate a release?#

Have the expert label about a hundred cases, with failures oversampled, and then see how many of those failures the judge caught. Also record how many good answers it passed, because raw agreement looks strong when most cases pass. Below a kappa of 0.4, check whether two people agree before you blame the judge, and split the question if they do not.

Sources and further reading

• Hamel Husain, Creating a LLM-as-a-Judge That Drives Business Results and the LLM Evals FAQ.
• Shankar et al. on criteria drift, via the same FAQ.
• Barnett et al., Seven Failure Points When Engineering a RAG System, IEEE/ACM CAIN 2024.
• Eugene Yan, Task-Specific LLM Evals that Do and Don't Work.
• User Feedback in Human-LLM Dialogues, EMNLP 2025, on what implicit signals can and cannot carry.
• Jason Liu, RAG series.
• Anthropic, Contextual Retrieval.
• Chroma, Evaluating Chunking Strategies and Context Rot.
• Ragas metrics for standard definitions.
• Ours: why RAG systems fail in production, why AI agents report unfinished work as done, when to fan out to many agents, and why temperature zero doesn't make an LLM agent repeatable.