The AI engineer roadmap
Stage 06 of 09Measure and improve

Evaluation and error analysis

The core production discipline. Not generic metrics, but looking at your own data, naming failures, and measuring the things that actually break.

error analysisLLM-as-judgeregression suites

Error analysis

Core

Read your system's real outputs and turn failures into a labelled taxonomy. The highest-leverage activity in AI engineering.

Concepts

Look at your dataReview 100+ real traces by hand; criteria can't be theorized, only discovered (Shankar).

The single highest-leverage activity in AI engineering is also the least glamorous: sit down and read your system's real outputs, one trace at a time. You cannot decide what to measure by theorizing in a meeting. The failure modes that actually matter only become visible once you have read a hundred or more real interactions. Teams that skip this step reliably build elaborate metrics that miss the failures their users actually hit.

Sources

Open and axial codingFree-label failures, then group them into a structured taxonomy.

A technique borrowed from qualitative research. First you open-code: read each failing trace and write a short, free-form label for what went wrong, with no fixed categories. Then you axial-code: cluster those labels into a structured taxonomy of failure modes. The output is a concrete, prioritized list of what your evaluators need to catch, grounded in evidence rather than guesswork.

Sources

Theoretical saturationKeep reading traces until new ones stop adding failure modes.

The stopping rule for error analysis. You keep reading and coding traces until new ones stop revealing new failure modes; at that point you have a representative picture and further reading has diminishing returns. In practice this is often around a hundred traces, but the real number is whatever it takes to reach saturation for your system.

Sources

Technologies

Annotation toolingA spreadsheet or trace viewer to label outputs fast.

The interface you use to review and label traces quickly. It can be as simple as a spreadsheet, but a purpose-built viewer that shows the full input, output, and context with one-keystroke labeling dramatically speeds up error analysis. Many teams build a small custom annotation app, because off-the-shelf tools rarely fit their data exactly.

Sources

Langfuse / BraintrustCapture traces and attach human labels in one place.

Platforms that capture production traces and let you attach human labels and datasets in one place, turning ad-hoc review into a repeatable workflow. They store the traces error analysis depends on and double as the home for your eval datasets and judge runs.

Sources

In production

The discipline that separates a shipped system from a demo.

Spend 60-80% hereMost of the work is understanding failures, not building automated checks (Husain).

A reframing of where the work is. Across experienced teams, most development effort on an AI product goes into understanding failures by looking at data, not into building automated checks or tweaking prompts. If you are spending the bulk of your time on anything else, you are probably optimizing the wrong thing.

Sources

Two experts, one verdictA good eval case is one where two domain experts independently agree pass/fail (Anthropic).

A quality bar for eval cases. A good evaluation task is one where two domain experts, judging independently, would reach the same pass/fail verdict. If they disagree, the criterion is ambiguous and no automated judge will do better; tighten the definition before you try to automate it.

Sources

Custom evaluators

Core

Build checks for your own failure modes. Generic metrics create false confidence.

Concepts

Binary failure modesDefine pass/fail checks tied to real failures, not vague quality scores.

Express each thing that can go wrong as a yes/no check tied to a specific failure you observed, not as a vague one-to-five quality score. Binary checks are easier to define, more consistent across raters, and directly actionable: a failing check points at a real, fixable problem rather than a number that drifted.

Sources

Assertions over judgesPrefer cheap deterministic checks where the property is code-checkable.

Whenever a property can be checked in code, check it in code. A regex, a schema validation, or a string match is cheaper, faster, and more reliable than asking a model to judge it. Reserve LLM-as-judge for the genuinely subjective criteria that no deterministic check can cover.

Sources

Application-specific criteriaYour evaluators come from your taxonomy, not an off-the-shelf metric list.

Your evaluators should come from your own failure taxonomy, not a generic metrics library. Off-the-shelf scores like 'helpfulness' or 'coherence' rarely map to what breaks in your product, and they create false confidence. The criteria that matter are the ones you discovered by reading your own data.

Sources

Technologies

PromptfooRun evals from config and fail the build on regressions.

An open-source tool for defining and running LLM evals from a config file, with built-in assertions and easy CI integration. It is a common way to turn a set of test cases into a gate that runs on every prompt or model change.

Sources

pytest assertionsPlain code checks for the deterministic properties.

For deterministic checks, your existing test framework is often all you need. Writing evals as ordinary pytest assertions means they run in the same CI pipeline as the rest of your code and fail the build the same way a unit test would, with no new infrastructure.

Sources

In production

The discipline that separates a shipped system from a demo.

Ditch generic metricsOff-the-shelf scores waste time and hide your real failures (Husain).

Prefab, general-purpose eval metrics waste time and hide your real failures behind a reassuring number. The prescribed alternative is error analysis first, then a small set of evaluators built specifically for the failure modes you actually found.

Sources

Regression-test every fixEach bug becomes a permanent case so it never silently returns.

Every time you fix a failure, add a case that captures it to your eval set. Over time this builds a regression suite that makes silent reintroductions impossible, the same discipline that bug-driven tests bring to traditional software.

Sources

LLM-as-judge

Core

Use a model to grade open-ended output, but only after aligning it to human labels.

Concepts

Calibrate to humansAlign the judge against 100+ human labels and measure agreement.

An LLM judge is only trustworthy once you have measured its agreement with human labels on your data. Collect a hundred or more human verdicts, run the judge on the same items, and check how often they match. If agreement is low, the judge's scores are noise dressed up as signal.

Sources

Full definition in the glossary
Binary over LikertBinary and pairwise verdicts are more consistent and higher-throughput than 1-5 scores.

Ask the judge for a binary or pairwise verdict rather than a one-to-five Likert score. Binary decisions are more precise, more consistent across raters, and higher-throughput; numeric scales invite arbitrary middle values that are hard to act on or align.

Sources

Escape hatchLet the judge answer 'Unknown' so it doesn't hallucinate a verdict (Anthropic).

Give the judge an explicit way to say 'Unknown' or 'cannot tell.' Without it, a judge forced to choose will hallucinate a confident verdict on ambiguous cases and pollute your metrics. The escape hatch also surfaces exactly the cases that need a human to look.

Sources

Per-dimension rubricsGrade each criterion with its own judge, not one judge for everything.

Grade one criterion at a time with its own focused judge, rather than asking a single judge to score everything at once. Isolated, single-purpose rubrics are more reliable and make it obvious which dimension failed when a case does not pass.

Sources

Technologies

Braintrust / LangfuseRun and track judge evals against datasets.

Eval platforms that run judge prompts against versioned datasets, track agreement with human labels, and let you compare judge versions over time. They turn LLM-as-judge from a one-off script into a maintained, measurable system.

Sources

In production

The discipline that separates a shipped system from a demo.

Few-shot the judgeGood and bad examples in the prompt are the strongest alignment lever (Shankar).

The strongest lever for improving a judge is putting a handful of good and bad labeled examples directly in its prompt. Few-shot examples align the judge to your specific preferences far more effectively than longer instructions. Newer rubric-decomposition methods compete, but few-shot remains a reliable first move.

Sources

Recalibrate on driftHuman preferences move; re-check judge agreement on a schedule.

Human preferences on LLM output change over time, a phenomenon called criteria drift. A judge that was aligned six months ago can quietly fall out of sync, so you re-check its agreement against fresh human labels on a schedule, not once.

Sources

Offline CI and online monitoring

Core

Two different jobs: catch regressions before merge, and watch live traffic after ship.

Concepts

Offline CI setsSmall, purpose-built datasets covering core features, past bugs, and edge cases.

Small, purpose-built datasets that run before merge, covering your core features, past bugs, and known edge cases. Because they run on every change, they favor cheap deterministic assertions over expensive judges; cost-per-run matters when the suite executes hundreds of times a day.

Sources

Online monitoringSample live traces and score them async with reference-free judges.

Evaluation of live production traffic, after ship. You sample real traces and score them asynchronously, leaning on reference-free evaluators like LLM-as-judge because there is usually no gold answer to compare against. It catches the failures your offline set never anticipated.

Sources

Eval sets from productionMine new cases from the bug tracker and support queue (Anthropic).

Your best eval cases come from real failures, not imagination. Mine your bug tracker, support queue, and monitored traces for examples that broke, and fold them back into the offline suite. This loop, from production failure to permanent test, is the engine of the data flywheel.

Sources

Technologies

CI pipelineRun the offline suite on every prompt or model change.

The same continuous-integration system that runs your unit tests should run your eval suite, gating prompt and model changes the way it gates code. Evals that do not run automatically on every change are evals that quietly rot.

Sources

LangfuseScore sampled production traffic continuously.

An open-source LLM engineering platform for tracing, prompt management, datasets, and online evals. It captures what each run did and lets you score sampled production traffic continuously. It can also act as an OpenTelemetry backend, so standards-based tracing does not lock you in.

Sources

In production

The discipline that separates a shipped system from a demo.

Block merges on evalsA prompt change that drops the suite does not ship.

Treat a drop in eval scores like a failing test: it blocks the merge. Making evals a hard gate is what turns them from a dashboard nobody reads into a real safety net for prompt and model changes.

Sources

Start with 20-50 casesAn eval set begins small and grows from real failures (Anthropic).

An eval set does not need to be large to be useful. Anthropic recommends starting with twenty to fifty tasks drawn from real failures; a small, high-quality set that everyone trusts beats a big, noisy one. It grows as you find new failures.

Sources

Tracing

Recommended

See exactly what a run did, so debugging is inspection rather than guesswork.

Concepts

Span-based tracingBreak a run into nested spans you can open and inspect.

Record each run as a tree of nested spans, one per model call, tool call, or retrieval step, with inputs, outputs, timing, and tokens on each. When something goes wrong you open the trace and read exactly what happened, turning debugging from guesswork into inspection.

Sources

Full definition in the glossary
Cost attributionSee token spend per step, per user, and per feature.

Because every span carries its token counts, traces let you see spend broken down by step, feature, and user. That is how you find the one prompt or the one agent loop quietly burning most of your budget.

Sources

Technologies

LangfuseTrace, replay, and compare individual runs.

An open-source LLM engineering platform for tracing, prompt management, datasets, and online evals. It captures what each run did and lets you score sampled production traffic continuously. It can also act as an OpenTelemetry backend, so standards-based tracing does not lock you in.

Sources

OpenTelemetryVendor-neutral spans across the whole run.

The vendor-neutral standard for distributed tracing, now with a GenAI semantic-convention extension for LLM spans. Emitting OpenTelemetry spans means your traces are portable: you can switch observability backends without re-instrumenting your code.

Sources

In production

The discipline that separates a shipped system from a demo.

Trace before you guessOpen the run and inspect the spans; debugging is reading, not guessing.

When an AI system misbehaves, resist the urge to theorize. Open the trace, read the actual inputs and outputs at each step, and the cause is usually obvious. Most 'the model is dumb' bugs turn out to be a bad retrieval or a malformed prompt, visible right there in the span tree.

Sources

Standardize on OTelEmit OpenTelemetry GenAI spans so you can swap backends later (Langfuse).

Instrument with OpenTelemetry GenAI conventions rather than a single vendor's SDK. You get the same rich traces today and the freedom to change tools tomorrow, because the data model is an open standard rather than a proprietary format.

Sources

ESG & sustainability

We shipped this layer in DocVerse AI.

Agentic search over dense ESG and institutional filings. It extracts the data, grounds every answer in a citation, fact-checks new documents against the knowledge base, and builds interactive reports on top.

Cited and fact-checked
Every answer
Read the case study

The vocabulary this layer assumes.

Concepts

LLM-as-a-Judge

In production

LLM Observability & Tracing

Put this whole roadmap on your team.

Every layer above is someone you can hire, production-proven and embedded in your team in days. Tell us what you are building and we will line up a shortlist.

Hire an AI engineerSee the work