Evals

An eval is a test suite for an LLM feature. It takes a set of inputs, runs your prompt or model against them, and scores the outputs so you can say, with evidence, whether a change made things better or worse. Without one, you are shipping on vibes: a prompt tweak that fixes one example quietly breaks five others, and nobody notices until a customer does. Evals turn "it feels better" into a number you can defend, and as of 2026 they are the clearest dividing line between teams that ship reliable AI features and teams that thrash.

Generic benchmarks do not predict your app#

Public leaderboards (MMLU, GSM8K, and the rest) measure abstract capability under conditions that look nothing like your task. Two models a point apart on a reasoning benchmark can swap rankings entirely on your invoice extraction or your support triage, and a frontier model that aces graduate-level science questions can still confuse two of your user personas or cite the wrong policy version. Many public benchmarks are also saturated or leaked into training data, which makes the remaining signal even weaker. As Hamel Husain puts it, prefab generic evals at best waste your time and at worst create an illusion of confidence that is not justified.

The signal you need comes from your own failures. The highest-leverage first step is unglamorous: pull up to a hundred real traces, label each one pass or fail with a short note about what went wrong, then cluster the notes into failure modes. A support assistant's clusters might look like: cited the wrong policy version (24 percent of failures), ignored a constraint the user stated (17 percent), promised a refund outside policy (9 percent). A useful stopping rule: when 20 consecutive traces surface no new category, you have found the dominant failure modes. Those clusters, not someone else's benchmark, define what you measure.

Binary checks beat graded scores#

For each failure mode, prefer a binary pass or fail check over a 1-to-5 score. Binary labels force clearer thinking and stay consistent across annotators, while Likert scales make a "3 versus 4" call subjective, invite raters to hide in the middle, and demand much larger samples to detect real differences.

There are two families of checker, and the order you reach for them matters. Code-based checks are plain assertions: did the JSON parse against the schema, does the cited document actually exist, is the proposed meeting time really on the calendar, does the output stay under the length limit. They run in milliseconds, cost nothing, and never change their mind, so exhaust them first; a surprising share of real failure modes (broken structured output, invented citations, ignored hard constraints) reduce to code. Model-based checks, known as LLM-as-judge, handle the fuzzy properties code cannot reach: tone, faithfulness to source material, whether the answer actually addressed the question. A judge is itself an LLM system, so it needs its own validation: have a domain expert label a sample of outputs, then iterate the judge prompt until it agrees with the expert at a rate you trust. Hamel Husain's published case study reached over 90 percent agreement with the domain expert in three prompt iterations. An unvalidated judge is just a second opinion you never checked.

Start with 20 to 50 cases from real data#

Teams stall for months waiting to build the perfect 500-case benchmark. Skip that. Anthropic's agent-evals guidance is blunt: 20 to 50 tasks drawn from real failures is a great start, because early in a system's life most changes move results by margins a small set detects easily. Build those cases from production: oversample traces with thumbs-down feedback, escalations to humans, and support tickets, since they are pre-sorted evidence of failure. Appoint one domain expert as the quality dictator rather than averaging three annotators into mush. Before launch, when you have no production data, a hand-written synthetic set is fine to bootstrap; replace it with real traces as soon as they exist, and let the suite grow toward a few hundred cases as regression coverage accumulates.

Wire evals into CI and block on regressions#

The payoff comes from treating evals exactly like tests. Every change to a prompt, a model version, or a retrieval setting runs the suite in CI, and a regression blocks the merge, because the whole point is that no change ships on a hunch. Two operational details matter. First, manage cost by tiering: run the cheap deterministic assertions on every commit, and reserve the judge-scored subset for nightly runs or pre-release checks. Second, read per-case diffs, not just the headline number: a suite that goes from 74 to 76 percent overall can still have broken the three cases that represent your largest customer. One caution from practitioners: do not chase a 100 percent pass rate. A suite you always pass is too easy; one hovering around 70 percent is actually stress-testing the system and leaving room to measure improvement. And pin the judge's model version, or your scores will drift for reasons that have nothing to do with your changes.

The eval flywheel#

A static suite goes stale as your product and your users change, so mature teams run a loop. Production observability captures full traces of real usage. Periodic failure analysis on fresh traces surfaces new failure modes, including ones no benchmark would have predicted, like users pasting screenshots of invoices instead of attaching files, or a hallucination pattern that only appears for one product line. Each novel failure becomes a new eval case, the fix lands, and the case guards against that regression forever. The loop compounds: after a year, the suite encodes everything the team has learned about how the system fails, which is why a model migration becomes an afternoon of running evals instead of weeks of manual QA. A competitor can copy your prompt in five minutes; they cannot copy the suite.

Practical takeaway#

Skip the leaderboards. Read your own traces, cluster the failures, and encode the top failure modes as 20 to 50 binary checks, code assertions first and validated judges only where code cannot reach. Wire the suite into CI so regressions block merges, keep it hard enough that you fail some of it, and feed every new production failure back in as a case. That loop, not any single score, is what makes an LLM feature steadily more reliable.