Dual-layer production agent ops: quality evals vs infrastructure investigation

How should a team operate an AI agent in production? We run it on two layers, each with its own owner and its own alerts. A quality layer scores a sample of real sessions, off the request path, to ask whether the agent did the job. An infrastructure layer watches every model call, tool call and permission check, and investigates when one fails. Checks that must never let a bad response through run inline on every turn. Both layers read the same trace, and every failure that escapes ends as an eval case or a probe.

The hard part is that agents fail without looking like they failed. A web service that cannot reach its database returns an error, and an alarm fires. An agent whose tool call is denied often returns a polite answer saying it found no records, with a success status, a finished trace and a user who now believes something false. Nothing in a standard service dashboard turns red, because by its measures nothing failed.

The second difficulty is that the two kinds of failure disguise themselves as each other. An infrastructure fault, such as a missing permission, produces an answer that reads like a quality problem: vague, unhelpful or empty. A quality fault, such as the wrong tool chosen for a request, produces clean infrastructure metrics, because every call succeeded. A team that watches only one layer misdiagnoses the other.

Ownership makes this worse, because the two layers usually belong to different teams. Quality usually belongs to the team that builds the agent, and infrastructure to platform or on-call engineers. Each looks at its own dashboard and sees a system that works. Without an agreed way to hand a failure from one layer to the other, it sits between them until a customer reports it.

This guide is about the operating architecture, not rubric design, which our guide on how to build LLM evals covers. It starts with why agents fail with a success status, then defines the two layers and what runs inline. It covers how each layer samples, scores and investigates, how the layers hand off, how a failure becomes an eval case without leaking personal data, and who owns each part.

Why do agents fail with a success status?

An agent turns errors into language. When a tool call fails, the model receives an error or an empty result and does what it was trained to do, which is write a helpful reply. The failure is absorbed into a sentence, and the status code reports that a sentence was successfully returned.

AWS gives a clear example in its 11 September 2026 write-up of monitoring a multi-agent airline booking system. The agents' execution role was missing permission to invoke the model. Every model call was denied at the IAM layer, and the supervisor agent returned a blank output rather than an error or an exception, because it had no model response to work with. From outside, the agent had stopped helping, and nothing reported a fault.

The first two rows are infrastructure failures that read as quality failures. The third is a quality failure that infrastructure cannot see. The last two sit between the layers, and they are the ones teams find latest. Our guide on why AI agents report unfinished work as done covers the last row in depth.

The fix starts in the tool layer, before any monitoring is added. A tool that receives an authorisation error should return a typed error, not an empty result. "Access denied for this account" and "no orders found" must be different outputs, because the model will treat them identically if they look identical. That one change moves a whole class of silent failures into the infrastructure layer, where they can raise an alarm.

What are the two layers of production agent ops?

The two layers ask different questions of the same run. The quality layer asks whether the user got what they needed. The infrastructure layer asks whether every call the agent made ran as intended. Neither answer implies the other, so each layer needs its own signals, cadence and owner.

This figure is a framework showing the split we use and the owners we assign, and it describes no vendor's product and reports no measured result.

The AWS reference architecture mentioned above draws the same line between the two layers. AgentCore Evaluations scores sampled live sessions on measures such as helpfulness, correctness and goal success. AWS DevOps Agent investigates infrastructure incidents by correlating logs, IAM policies and orchestration traces across services. The post describes the two as showing whether the agent works correctly and whether the infrastructure supports it, which are our two questions.

We keep the layers separate even when one tool could serve both, because they fail at different speeds and page different people. A quality regression is a pattern across many sessions, found by reading scores over hours. An expired credential is one fault affecting every session at once, and it needs a page within minutes. One alert policy cannot serve both without waking people for noise or missing the outage.

What runs inline, and what runs later?

Inline checks run on every turn, before the response reaches the user, and they can block it. Everything else runs later, on a sample or on a signal. The rule we use is short: a check goes inline if a failure must never reach a user, and if it can be decided quickly and exactly enough to block on.

This figure is a framework, and the checks in each column are the ones we usually place there. Your own list depends on what your agent is allowed to do.

Sampling means some bad responses reach users before anyone scores them. AWS says as much in its own post, noting that because online evaluation runs asynchronously on a sample, a problematic response can reach the user first. It recommends inline guardrails for content filtering, denied topics, grounding checks and personal data redaction. We apply the same test to every check: if the harm happens on delivery, scoring it an hour later only tells you it happened.

Every inline check spends latency on every turn, so the inline list has to stay short. We keep inline checks in code wherever the rule can be written down, and use model-based guardrails only where it cannot, each with a timeout. A model-based check that times out needs a decision already made in code: block and hand over, or let the response through and flag it for review. Leaving that choice to the moment of failure means the system decides by accident.

How should the quality layer sample and score?

The quality layer scores a sample chosen on purpose, not a random slice of traffic. Random sampling spends most of its budget on the common, easy sessions that already work. We stratify, so rare routes, new versions and sessions that ended badly are scored far more often than their share of traffic.

Some sessions are always scored, whatever the sampling rate. A session where the user reversed an action, complained, asked for a person or repeated the same request is scored every time, because each is a signal from the user that something went wrong. The sample then covers each route and tool, each customer segment that matters, and each new prompt or model version at a higher rate for its first days in production.

The scoring itself follows the rubric work in our evals guide. We write binary criteria from real failures and use a model judge only where a rule cannot decide. Each judge is checked against people before its scores count, and the judge alignment entry covers how. AWS's own caveat applies: LLM-based scores lack ground truth, and should be read as signals and calibrated with subject matter experts.

Scores are an input, and patterns are the output, since a single low score means little on its own. A cluster of low scores that share a route, a tool or a phrasing in the request is a failure mode with a likely cause. We group low-scoring sessions by those fields every day, and each group with a clear cause becomes a work item with an owner.

Keep the quality layer's alerts slow and specific. An alert on a daily drop in one route's score, with the sessions attached, gets read. An alert on every low score gets muted within a week, and then a real regression passes unseen.

How should the infrastructure layer investigate?

The infrastructure layer watches every call and investigates when one fails. It needs each hop of an agent run recorded as a span with the same trace ID: the model call, each tool call, each retrieval, each permission check, and any retry or fallback. Without that, an investigation starts from a blank reply and works backwards through several services' logs by hand.

The spans need fields that standard web tracing does not carry. We record the permission decision and the principal it was made for, the tool's result status as a typed value, whether a result was empty, how many retries ran, and which model actually answered when a fallback fired. Those fields turn "the agent said nothing useful" into "the model call was denied for this role".

Investigation correlates those signals across the services a run touched. AWS DevOps Agent is one tool that does this automatically, pulling logs, building a topology of affected resources and tracing the failure path. In the AWS example it connected blank agent responses to the missing model permission. A team without such a tool needs the same result from a runbook: start at the first failing span in the trace and follow it to the service that owns it.

Every investigation ends with a probe. Once the cause is fixed, we add a synthetic run that exercises the failing path on a schedule, and an alarm on the signal that was missing. A probe that calls the agent as a real user would, with a known expected result, catches the next expired credential before a user does.

How do the two layers hand off to each other?

The layers share a trace ID, and the handoff between them runs on it. A quality score that points at a session also points at every span in that session. An infrastructure alarm that names a hop also names the sessions that passed through it. Harness's AgentTrace documentation describes the same design, with each eval score carrying the trace ID of the run it scored.

Three rules keep the handoff from becoming an argument. A quality drop in the same window as an infrastructure alarm goes to infrastructure first, because a broken dependency explains many quality symptoms at once. A quality drop concentrated on one tool gets a health check on that tool before anyone edits a prompt. After an infrastructure fix, the quality layer rescores a sample of the affected sessions, so both owners can see the symptom has gone.

The misdiagnosis we most want to prevent is a prompt change made to fix an infrastructure fault. It seems to work, because the fault often clears on its own, and it leaves behind a prompt tuned to a failure that no longer exists. Checking infrastructure first costs minutes. Skipping it can cost a week of prompt edits and a regression nobody can explain.

How does a production failure become an eval case?

Every escaped failure should end as a check that would have caught it. A quality cause becomes an eval case that gates releases, and an infrastructure cause becomes a probe and an alarm. The loop only works if cases can leave production without carrying personal data with them.

This figure is a framework of the steps we run, and it carries no timings or counts.

Redaction comes first, before the case leaves the production environment. Names, contact details, account numbers and free-text personal details are replaced with consistent placeholders, so the same customer gets the same token everywhere in the case. The structure of the failure survives and the person does not. Where the failure depends on the content of a personal detail, such as a name the agent misread, we write a synthetic replacement with the same property rather than keep the real one.

Reproduction comes next, shrinking the session to the smallest case that still fails. A long session with one bad tool choice near the end becomes the few turns of context that lead to that choice. Small cases run fast in CI, fail for one reason, and do not break when an unrelated part of the conversation changes.

Each case is labelled with the expected behaviour and an owner. The label says what a correct run does, such as calling the lookup tool before answering, or handing over when the account is locked, rather than pasting in one ideal reply. The owner is the person who decides whether a future failure of this case blocks a release.

The case then joins the eval set, and the set gates releases. AWS's 8 September 2026 walkthrough of running AgentCore evaluations from GitHub Actions shows the shape, with a pull request that fails when the agent's score drops. Harness describes one-click export of a production failure into an eval dataset. The tooling varies, and the rule does not: a failure that reached a user once should fail the build next time.

Keep the raw session out of the repository even after redaction review. Store cases in a controlled dataset with access rules, a retention period and a link back to the production trace for people permitted to see it. A test fixture copied into source control is copied into every clone and every fork.

Who owns what?

Each part of the system needs one named owner, and the owners need to agree on how a failure moves between them. Many operating failures are ownership failures. The signal existed, and it reached nobody who could act, or it reached two teams who each assumed the other had it.

We hold one short review a week across the owners. It covers the week's escaped failures, which lane each went down, and whether each ended in a case or a probe. Our guide on building an audit trail for AI decisions covers the decision records this review reads. Our AI observability work sets this up alongside the agent, so the two layers exist before the first incident rather than after it.

The release model ties this back to autonomy and to how much an agent may do. Our guide on rolling out AI agent autonomy in five levels covers how an action earns more authority from evidence. The quality layer is where that evidence comes from, and the infrastructure layer keeps it honest, because a quality score measured on a system with a silent tool failure is measuring the wrong system.

How do you start with a small team?

A small team can build both layers in stages, and the order matters more than the tooling. Each step makes the next one possible, and the first two cost little but change what every later signal means.

1. Put one trace ID on every hop of every run, and record permission decisions, typed tool status and empty results on the spans.
2. Make tools return typed errors, so a denied call and an empty result can never look the same to the model or to a dashboard.
3. Write down the must-never list and move each item inline, in code where possible.
4. Score every session with a user signal, then add a stratified sample for one route.
5. Run the failure loop on every escaped failure, with redaction first, and hold the weekly review.

A team that starts with a quality dashboard and skips the first two steps can spend weeks tuning prompts against infrastructure faults, because nothing tells them a tool was denied. The trace and the typed errors make the rest of the work diagnosable.

Common questions

What is the difference between agent evals and agent observability?#

Observability records what the agent did: every model call, tool call, permission check and result, as spans in a trace. Evals judge whether what it did was good, by scoring sessions against criteria. Production agents need both, read from the same trace. Observability alone cannot tell you a fluent answer was wrong, and evals alone cannot tell you a tool was denied.

Why does an AI agent return a success status when it has failed?#

Because the agent turns errors into language. When a tool call is denied or returns nothing, the model writes a reply anyway, and the service reports that a reply was returned. The fix starts with tools that return typed errors, so a denied call and an empty result look different to both the model and the monitoring.

Which checks should run inline on every agent turn?#

Checks where the harm happens on delivery or execution, and which can be decided quickly and exactly. That covers tool argument validation, permissions and allowlists, personal data redaction, denied topics and regulated wording, and a check that cited sources exist. Judgments about correctness and tool choice run later on a sample.

How much production traffic should quality evals sample?#

There is no single right share, and a random share is the wrong shape. Score every session with a user signal such as a reversal, complaint or handoff. Then sample by stratum, covering each route, tool and important segment, and oversample new versions for their first days in production. Tune the rate by whether it finds failure patterns, not by habit.

Who should own AI agent quality and infrastructure in production?#

The agent team owns quality: prompts, tool selection, evals and release gates. Platform or on-call engineering owns infrastructure: credentials, quotas, dependencies and tracing. A product or business owner decides what counts as correct, and security and privacy own redaction and access. The owners share one trace and a weekly review of escaped failures.

How do you turn a production failure into an eval case without leaking personal data?#

Redact before the case leaves production, replacing personal details with consistent placeholders, or with synthetic values where the detail matters to the failure. Shrink the session to the smallest failing case and label the expected behaviour and owner. Store it in a controlled dataset with access rules and retention, never in source control.

How is this different from building an eval rubric?#

A rubric decides how to score one session. This guide covers the operating system around the rubric: what runs inline, what is sampled, how infrastructure failures are found and fixed, how the layers hand off, and how failures become cases. Our guide on how to build LLM evals covers writing the rubric itself.

What should trigger an infrastructure investigation for an agent?#

A rise in denials or errors on one hop, empty results from one tool without a change in traffic, blank model responses, a rising share of fallback model calls, more retries per session, or latency up on a single hop. Start at the first failing span in an affected trace and follow it to the service that owns it.

Further reading

• How to build LLM evals. Writing the rubric and validating judges, which this guide does not repeat.
• Why AI agents report unfinished work as done. Expected-state checks for the failures that sit between the layers.
• How to build an audit trail for AI decisions. The decision records the weekly review reads.
• How to roll out AI agent autonomy in five levels. How the evidence from both layers moves an action's authority up or down.
• How to scope AI agent permissions. The permission decisions the infrastructure layer records on each span.
• LLM as judge and judge alignment. Glossary entries for scoring with a model and checking it against people.
• AWS, Monitoring production agent lifecycle with AWS DevOps Agent and AgentCore Evaluations, 11 September 2026. The dual-monitoring reference architecture and the missing-permission example.
• AWS, Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions, 8 September 2026. An eval gate that fails a pull request on regression.
• Harness, AgentTrace: an observability and guardrail framework. Eval scores joined to traces by ID, and production failures exported as eval cases.
• Harness, harness-evals. The open-source evaluation layer, with importers from OpenTelemetry traces.