Evals & LLM Evaluation Metrics

LLM evals are test suites for software built on a language model. An eval runs a fixed set of inputs through your prompt or model, scores each output, and tells you whether a change made things better or worse. Those scores are your LLM evaluation metrics, and the useful ones come from your own failures rather than from a standard list. A unit test checks that a function returns four. An eval checks things that need judgement: whether an answer is supported by its sources, whether a citation says what the claim says, whether the system declined a question it had no evidence for. Without one, a prompt edit that fixes the example in front of you can break five others, and nobody finds out until a customer does.

Why public benchmarks do not predict your product#

Public leaderboards measure general capability on tasks that look nothing like your invoices or your support queue. Two models a point apart on a reasoning benchmark can swap places on your task. Benchmarks also run out of headroom: Anthropic's January 2026 post on agent evals notes that scores on SWE-bench Verified, a coding benchmark, climbed from around 30% to over 80% within about a year, so it barely separates the strongest models any more. Husain and Shankar's evals FAQ puts it more bluntly: generic evaluations "waste time and create false confidence when you use them as quality measures."

The cases that matter come from your own failures. Read real traces, at least 100 of them by the FAQ's advice. A trace is the logged record of one request: what came in, what was retrieved and what the model said. Write a short note on each one that went wrong. Then group the notes into failure modes and count them. Our guide to building evals stops reading when 20 traces in a row add no new kind of failure. The counts tell you where to spend the next month, because a mode that causes four failures in ten deserves more work than one that causes one in fifty.

Checks in code first, a model judge second#

Write one check per failure mode, and make it pass or fail. A 1 to 5 score asks two raters to agree on the difference between a 3 and a 4, and they rarely do, while a binary label forces the decision that the scale lets people dodge.

Many failure modes turn out to be checkable in code. Did the output parse against the schema? Does the cited document exist? Is the proposed meeting time free on the calendar? These checks run in milliseconds, cost nothing per run and give the same answer every time, so write them before anything else. On our trademark drafting platform, a validator rejects any citation that does not resolve, and no judge is needed for that.

The rest need reading comprehension: tone, whether the answer is faithful to its sources, whether it answered the question asked. For those you use LLM-as-judge, a second model prompted to grade the first. A judge is itself a model and gets things wrong, so it has to earn trust. Have a domain expert label a sample, and revise the judge prompt until it agrees with them. Honeycomb's query assistant got its judge past 90% agreement with the expert in three revisions, and raw agreement was a fair test there only because about half the cases were failures. When failures are rare, report how many real failures the judge catches and how many good outputs it passes, separately, because a judge that passes everything can still agree with the expert most of the time.

How many cases you need to start#

Fewer than most teams think, and teams that wait for a large set tend never to start. Start with 20-50 cases drawn from real failures, the range Anthropic's post calls a great start, because early changes to a system tend to have large effects that a small set can see. Take the cases from production where you can, such as traces a user rated badly, conversations handed to a person and support tickets, because each is a failure someone already found. Before launch, hand-written cases will do, and real traces replace them as they arrive.

Running the suite on every change#

Run the suite on every change to a prompt, a model version or a retrieval setting, and block the merge when it regresses. Read the cases that flipped, not only the total. A suite can rise from 74 to 76 percent overall while breaking the three cases that belong to your largest customer.

Do not aim for 100 percent. The FAQ's point is that a suite you always pass is not testing much, and a pass rate nearer 70% leaves room to see an improvement. Pin the judge's model version too, because if the judge model changes underneath you, its scores move for reasons that have nothing to do with your change.

The suite also has to grow. New failure modes appear as users find new ways to use the product, so each production failure you diagnose should come back as a case. On our DevOps agent, the validation loop records every failure and revision with the request that caused it, and the team treats that record as the input for improving the loop rather than the prompt. We measured template errors before deploy falling from about 10% to 4% once the loop went in.

Which LLM evaluation metrics to track#

Choose LLM evaluation metrics from your failure modes, not from a list. The headline number is the pass rate of each pass-or-fail check, one per failure mode, read case by case so you can see which ones flipped.

A few failure types have a standard metric with its own page here. For a RAG answer, groundedness checks that every claim is supported by the passages retrieved for it. For the retrieval underneath, recall@k and NDCG say whether the right passage arrived and how high it ranked. For speed, time to first token and time between tokens need separate budgets, read at percentiles. For the judge itself, report how many real failures it catches and how many good outputs it passes, with Cohen's kappa beside them, because raw agreement hides a judge that passes everything.

When a scale is the right score#

Binary checks tell you whether a release is safe, and they are less useful for comparing two models on the same work. The trademark drafting platform scores every draft from 0 to 10 on six named criteria before an attorney sees it: structure, reasoning depth, citation accuracy, jurisdiction discipline, edge-case handling and tone. Production logging scores the base and fine-tuned models on the same inputs, so a regression after a model change shows up as a score that moved. The attorney's corrections still matter most, because they are the one measure of what automated scoring cannot see. Use binary checks to gate a release, and add a scale with named criteria only when the question is whether one model beats another on the same inputs.