Building production-grade analytics agents

How do we build a production-grade analytics agent? We build it as a layered system that turns a business question into a governed plan, the plan into SQL, the SQL into checked rows, and the rows into a cited answer. A metric layer owns every definition, gates parse and price each query before it runs, and a bounded workflow runs the steps. A trace records each step, so any answer can be explained and replayed later.

Jev, TypeSafe's decision model, judges the edges where a gate trips or a case is ambiguous, and code carries out each decision. People review the cases that earn their time, such as a first-time metric ambiguity or a high-cost query, and stay off the routine SELECT paths the gates already cover. The rule throughout is that Jev judges, code enforces, and the model generates.

The hard part is that an analytics agent fails quietly. A support agent that misreads a ticket usually produces a reply someone can see is wrong. An analytics agent that joins orders to line items without aggregating first produces a revenue figure that is too high, formatted like every other revenue figure, with a chart to match. Nobody objects until the number reaches a board deck and disagrees with finance.

The second difficulty is that the unsafe outputs are not all obvious. A DELETE statement is easy to catch. A SELECT that scans three years of events because the model dropped a date filter is valid SQL, runs under a read-only role, and still costs real money and slows every dashboard sharing the warehouse.

A system that answers correctly today can also drift without anyone noticing. A metric definition changes, a prompt is edited, or a new table appears, and an answer that was right last month is now wrong with nothing in the logs to say why. Production-grade therefore includes the parts that make an answer explainable and reproducible later: traces, decision logs, versioned definitions and change control.

This guide defines production-grade first, then sets out the layers of the system. It goes deep on the minimum technical core: SQL generation, AST parsing, the EXPLAIN gate, the database guardrails, the answer filter, and Jev as the decision layer, following our guide on introducing Jev into agentic workflows. It then covers orchestration, auditability and governance, where people belong in the loop, and the evals and release stages that tell you whether the system works.

What does production-grade mean for an analytics agent?

Production-grade means the agent's answers can be trusted, afforded, explained and changed safely, and each of those needs a named part of the system. A demo that answers ten questions correctly against a sample database proves the model can write SQL. It proves nothing about the eleventh question, the real data volume, or what happens when finance renames a metric. The table sets out the properties we hold every build to.

Correctness is the hardest of these to hold, because an analytics agent's worst output is a query that runs and returns a wrong number. Most agent failures end in an error, a refusal, or a visibly odd action. Here the warehouse accepts the query, returns rows, and the model writes a fluent paragraph around them. The table sets out the failures we design against and the gate that catches each one first.

Most rows in that table pass every check a general agent harness would run. The SQL is well formed, the tool call succeeded, and the answer reads well. We build the stack below around one question that general checks do not ask: would the number survive an analyst rerunning it by hand?

That question also explains why the gates sit where they do. The expensive failures happen at the boundaries between steps, when a vague question becomes a precise query, when a query meets real data volumes, and when rows become a sentence. Each gate checks one boundary, and each gate's output is something a person can read later.

What are the layers of a production analytics agent?

A production analytics agent has ten layers, and a question passes through seven of them in order. The other three, the metric layer, memory and the tool boundary, serve every step instead of sitting at one point on the path. We draw the layers separately because each one fails in its own way, and each needs its own check and its own record in the trace.

This figure is a framework of the layers we build. It describes no product's internals and carries no measured values.

The table gives each layer's job and the failure it exists to prevent. The six layers on the query path that carry most of the risk get their own sections after this one, and the rest are covered here.

Intake and intent#

Intake decides what kind of question arrived before anything is planned. A definition question, such as how active users are counted, needs the metric layer and no query. A lookup against a governed metric needs one compiled query, and an open analysis needs a plan with several steps. Intake also resolves the entities and dates, so "last quarter" becomes explicit dates in the fiscal calendar and "EMEA" becomes the region codes the warehouse uses.

Ambiguity detection pays for itself fastest. When a question could mean two governed metrics, such as bookings or recognised revenue, the agent asks which one and records the answer for that team. Guessing produces a number that is right for one reading and wrong for the other, and nothing in the answer tells the reader which reading was used.

Planning#

Planning turns the resolved question into typed steps, each naming the metrics, dimensions, filters and time window it needs. A one-step plan covers most lookups. A question about why churn rose in September becomes several steps: churn by month, churn by segment around the change, and each segment's share of accounts. Each step is small enough to check on its own, and the plan is the record a reviewer reads first when an answer is wrong.

This is the plan-and-execute pattern applied to analysis, and it gives every later layer a reference point. The gates check SQL against its step, the filter checks the answer against the plan's scope, and the orchestration section below covers how steps run and recover.

Result shaping#

Result shaping sits between the warehouse and the model, and it decides what the model may read. The model does not need ten thousand rows to describe a trend. Code aggregates or samples results to what the answer needs, masks columns tagged as personal data, and passes a summary with row counts, ranges and null counts alongside the shaped rows.

Keeping raw rows out of the model's context limits what a logged completion or a leaked prompt can expose. It also keeps a long result from crowding the question and the plan out of the context window, which is a common cause of answers that describe the data correctly and answer the wrong question.

Answer synthesis#

Synthesis writes the prose, and usually a chart specification, from the shaped result and the plan. The model writes the words and picks a chart type. Code computes every derived number, such as a percentage change or a share, because arithmetic has one right answer and a model can get it wrong. The chart is rendered from the result set, never from numbers the model typed, so the chart and the table cannot disagree.

The draft then goes to the answer filter covered below, which checks every number and claim against the rows before anyone reads it. Separating synthesis from the filter matters because the model that wrote a claim is poorly placed to judge whether the rows support it.

Memory and context#

Memory in an analytics agent is mostly scope. A follow-up such as "and for enterprise only" inherits the metric, time window and filters of the previous question, and the agent should carry them explicitly in state, not rely on the model rereading the transcript. We store each turn's resolved plan and its results by handle, so a follow-up can refer to a result without pasting its rows back into the prompt.

Every answer states its full scope, so a filter carried over from three turns ago is visible to the reader. Longer-lived agent memory, such as a team's answer to a metric ambiguity or a saved analysis, belongs in the governed layer with an owner. A preference the agent learns silently and applies to later answers is a definition change nobody reviewed.

Tool and MCP boundaries#

Other agents and tools reach the analytics agent as well as people, and its boundary decides what they can ask for. We expose typed tools through MCP, such as asking a metric question, explaining a definition and running a saved analysis, and every call runs as the calling user. Saved analyses can be published as playbooks the calling agent searches, which our guide on agent playbooks as MCP tools covers.

We do not expose a tool that runs arbitrary SQL. Our guide on MCP tool design sets out why a raw SQL tool is one tool in name and every operation in practice. For analytics it also lets a caller skip the planning, the metric layer and the answer filter, and get rows with none of the checks this guide describes.

How should the agent generate SQL?

SQL generation opens the minimum technical core, the six parts we build before anything else and go deep on here: generation, parsing, the EXPLAIN gate, the database guardrails, the answer filter and Jev. The agent should write a plan before it writes SQL, and the plan should name metrics and dimensions from a governed layer. The model then picks from definitions that already have an owner, instead of inventing a definition of revenue in each prompt.

A plan is a small structured output: the metrics, the dimensions to group by, the filters, the time window and grain, and the dialect. Code validates it against the metric layer, and for questions the layer covers, the layer compiles the plan to SQL, so the model never writes the join logic behind a governed metric.

{
  "metrics": ["net_revenue"],
  "dimensions": ["region"],
  "filters": [{ "field": "order_status", "op": "in", "values": ["completed"] }],
  "time": { "field": "order_date", "from": "2026-07-01", "to": "2026-09-30", "grain": "month" },
  "dialect": "snowflake"
}

The plan gives every later gate something to check against. The parser can confirm the SQL uses the plan's tables, and the answer filter can confirm the answer states the plan's time window. A reviewer reading a wrong answer can also see whether the plan was wrong or the SQL drifted from a correct plan, which are two different fixes.

Questions outside the metric layer still come up, and for those the model writes SQL directly. We give it a narrow slice of the schema for that question, never the whole catalogue, and lock the dialect so the model cannot mix syntax from two warehouses.

Constrained decoding can force the plan into its schema at generation time. It cannot make free SQL correct, which is why free SQL goes through the same parser and runs at a lower level of autonomy. A wrong plan against the layer fails in a way a reviewer can read, while wrong free SQL can look like any other query.

If you do not have a metric layer yet, the agent will build one implicitly in its prompts, one inconsistent definition at a time. Our guide on shared business context for every AI agent covers defining terms once, with an owner and a version, and serving them to every agent. An existing semantic layer is a good source for those definitions, and the agent should use it rather than copy it.

Why parse the SQL before it runs?

Parse the SQL into an abstract syntax tree before it reaches the warehouse, because a string check cannot tell you what a query does. A regular expression looking for DELETE misses a comment trick or a keyword hidden in a string literal, and it flags a column named deleted_at. A real parser for the locked dialect, such as sqlglot in Python, gives code a tree of statements, tables, columns, joins and clauses to check. The table lists the checks we run and what each one stops.

The parser may make mechanical fixes, such as adding a missing LIMIT or qualifying a table with its schema. It never rewrites meaning. If it changed a join key or added a filter, the query that ran would no longer be the one the model reasoned about, and the answer would describe a query nobody wrote. A rejection goes back to the model with the failed check named, and the number of retries is capped in code.

The sketch below shows the shape of the check. It is schematic, and a production version also resolves aliases and subqueries before checking columns.

import sqlglot
from sqlglot import exp
 
WRITE_NODES = (exp.Insert, exp.Update, exp.Delete, exp.Merge, exp.Create, exp.Drop, exp.Command)
MAX_ROWS = 1000  # a named constant, reviewed like code
 
def check_sql(sql: str, dialect: str, allow: Allowlist) -> Verdict:
    try:
        statements = sqlglot.parse(sql, read=dialect)
    except sqlglot.errors.ParseError as e:
        return Verdict.reject("parse", hard=False, detail=str(e))
    if len(statements) != 1:
        return Verdict.reject("multi_statement", hard=True)
    tree = statements[0]
    if not isinstance(tree, exp.Select) or any(tree.find_all(*WRITE_NODES)):
        return Verdict.reject("not_select", hard=True)
    for table in tree.find_all(exp.Table):
        if not allow.table(table.db, table.name):
            return Verdict.reject("table_not_allowed", hard=True, detail=table.sql())
    for join in tree.find_all(exp.Join):
        if not join.args.get("on") or not allow.join_keys(join):
            return Verdict.reject("join_keys", hard=False, detail=join.sql())
    if not tree.args.get("limit"):
        tree = tree.limit(MAX_ROWS)
    return Verdict.ok(tree.sql(dialect=dialect))

Notice the hard flag on each rejection. A write statement, a second statement or a table off the allowlist is a hard fail, and code denies it without asking anything. A join on an undeclared key is a soft fail, because some are legitimate, and the Jev section below covers how soft fails become decisions. Keeping the two classes separate in code means a clever model argument can never talk its way past a hard fail.

What should an EXPLAIN gate check?

An EXPLAIN gate asks the warehouse how it plans to run the query, then blocks the query if the plan is too large, before the real query runs. The parser checks the shape of the SQL. It cannot know that a well-formed query over one table will read four terabytes, because that depends on data volumes, partitioning and statistics only the warehouse has. Each warehouse exposes this differently, and the gate reads whichever the warehouse offers.

The ANALYZE warning in the first row is not a detail. In PostgreSQL, EXPLAIN ANALYZE executes the query to measure it, so a gate built with it runs every query it was meant to screen, including the ones it then rejects. Test the gate against a query that would fail loudly if it ran, and confirm it never does.

From the plan, the gate checks four things against budgets that live in code as named constants. It checks estimated bytes or cost against a per-question budget, and whether a large table is being scanned in full. It checks the estimated output rows against the row cap, and whether a join's estimated output is far larger than its inputs, which is how fanout shows up in a plan.

Each check has two limits. A hard budget denies the query outright, and a lower warning line marks the middle band where a judgment is needed. Queries between the two lines go to the decision step described in the Jev section below, with the plan attached as evidence.

Estimates can be wrong. Stale statistics make a planner guess low, and some warehouses estimate bytes before pruning partitions. We treat EXPLAIN as a pre-check that stops most runaway queries early and cheaply, and keep the database's own limits as the guarantee. A query that slips past a bad estimate still hits the timeout and the byte cap.

Which guardrails belong in the database?

Put every limit you can in the database itself, because those limits still hold when the agent's code has a bug. The parser and the EXPLAIN gate are our code, and our code can be wrong. A read-only role with grants only on reviewed views stops a write even if a parser upgrade changes how it classifies a statement. The table sets out the controls we configure before the agent runs a single query.

We also pin the connection to one dialect and one warehouse account, so a prompt cannot redirect the agent to a different target. The agent never holds a credential with more access than the role above, and its queries run with the asking user's access, so it cannot show a sales manager rows the manager could not query directly. The governance section below covers how those roles are scoped and who approves changes to them.

Dry-run mode deserves more use than it gets. A new question class can run for a period with the full stack active and execution switched off. The logs then show what the parser and EXPLAIN gate would have done, with no risk that any of it executed.

What should filter the answer before a person reads it?

The answer filter checks the drafted answer against the query and the rows it returned, and blocks or rewrites anything the rows do not support. Rows are not an answer. The model still has to describe them, and in that step it can round a number differently, state a time window the query did not use, or describe a trend the chart does not show. The filter runs after results and before the reader, in two layers.

The first layer is deterministic, and it runs in code. Personal data columns flagged in the catalogue are redacted from rows before the model sees them. Small groups can be suppressed so an aggregate does not identify one person. Every number in the draft is matched to a cell in the result set, and a number with no source cell blocks the answer. An empty result gets a fixed message saying no rows matched, which prevents the model from reporting it as zero.

The second layer judges meaning, which code cannot do alone. Does the draft claim a cause, a trend or a comparison the rows do not show? Does the stated scope, such as the period, region and segment, match the executed SQL? Does the answer address the question asked, or a nearby one the metric layer happened to cover? These are bounded judgments with yes or no answers, which is the shape of question Jev handles, and the next section puts them there.

Every answer that passes carries its evidence with it. That means the SQL as executed, the metric definitions and their versions, the row count, the data freshness, and a query ID that links to the log. A number with no query behind it does not reach the reader, and groundedness stops being a score on a dashboard and becomes a condition for sending the answer.

When the filter cannot make the draft grounded, the agent abstains instead of sending its best guess. An abstention names what is missing, such as a metric the layer does not define or a time window the data does not cover. A reader can act on that, and cannot act on a confident guess that happens to be wrong.

How does Jev sit over the gates?

Jev sits beside the pipeline as the judge on its decision edges, and never inside the execution path. Our guide on introducing Jev into agentic workflows sets out the rule we apply here unchanged: Jev judges, code enforces, and the model generates. The model writes the plan, the SQL and the prose. Code runs the parser, the EXPLAIN gate, the database controls and the deterministic filter, and decides every outcome. Jev answers typed questions with calibrated probabilities where a gate needs a judgment.

This figure is a framework showing where each check lives and who decides. It describes no product's internals and carries no measured values.

The five steps from the Jev guide map onto the analytics stack in the same order. Each step gives Jev's answers more influence, and each one produces the logs the next is calibrated against.

The needs-a-tool question matters more here than in most agents. Many analytics questions are about definitions, such as how active users are counted, and the metric layer answers them without touching the warehouse. Without the Noul, a Choice over datasets always picks one, and the agent runs a query to answer a question that needed a sentence.

How do gate trips become Jev decisions?#

A gate trip is a fact produced by code, and the decision about what happens next depends on its class. Hard fails never reach Jev. Soft fails and warning-band results go to Jev as state, together with the question, the plan and the gate's findings, and code turns Jev's answer into one of three outcomes: deny, escalate to an analyst, or allow with an audit record.

The code that applies these outcomes is short, and the thresholds are named constants in it. As in the Jev guide, the client call is schematic.

WARN_ALLOW_MIN = 0.90   # examples, set from your own shadow logs
WARN_DENY_BELOW = 0.40
MAX_AUTO_RADIUS = 2
 
def decide(trip: GateTrip, ctx: QueryContext) -> Outcome:
    if trip.hard:
        return audit("deny", trip, ctx)
    answers = jev.evaluate(
        state={"question": ctx.question, "plan": ctx.plan, "sql": ctx.sql, "findings": trip.findings},
        questions=QUESTIONS[trip.kind],  # fixed text, versioned
    )
    p_ok = answers["proportionate"].p_yes
    radius = answers["blast_radius"].score
    if radius > MAX_AUTO_RADIUS or p_ok < WARN_ALLOW_MIN:
        outcome = "deny" if p_ok < WARN_DENY_BELOW else "escalate"
    else:
        outcome = "allow_with_audit"
    return audit(outcome, trip, ctx, answers=answers)

Two details in the sketch carry most of the safety. The user's question, the plan and the SQL sit in named state fields, and the question text is fixed and versioned. A question typed by a user, or text stored in a table the agent reads, can carry prompt injection aimed at the judge, and keeping it out of the question text limits what it can rewrite.

Every Jev call is logged with the full probability distribution, the question versions and the thresholds in force. When someone later asks why a warning-band query ran, the log answers with a probability, a threshold and the plan Jev saw. That record is one entry in the per-question trace the auditability section below describes.

Some things stay in code however good Jev's answers get. That covers the read-only role and every grant, the allowlists, the byte caps and timeouts, and the arithmetic that matches numbers to cells. It also covers the thresholds, the retry limits, and the assembly of the state Jev reads. A calibrated 0.95 on a scan being proportionate is still wrong one time in twenty, so a budget Jev could raise would stop being a budget.

Should the agent run as a workflow or a free loop?

Run it as a workflow with bounded steps, and give the model freedom only inside each step. A free agent loop, where the model picks the next tool at every turn, is good at open exploration and poor at producing the same answer twice. The same question asked twice should run the same plan, hit the same gates and return the same number, so the outer shape of an analytics agent is fixed code.

We allow a free loop in one place, inside an analysis an analyst is running with the agent as an assistant. Even there, every query goes through the same gates and limits, and the analyst sees each step. The figure shows the workflow shape we use for everything else.

This figure is a framework of how a planned question runs. It carries no limit values, because those come from each team's budgets and logs.

Plan, then execute#

The planner writes the whole plan before any step runs, and code validates it against the metric layer and the budgets. A plan that needs twelve queries for a simple lookup is rejected before it spends anything. Steps then run in order, or in parallel where they do not depend on each other, and each result is stored by handle for later steps and for synthesis.

When a step's result changes what the rest of the plan should do, the orchestrator asks the model for a revised plan and validates it again. Letting the model improvise the next query in the middle of a run would put an unplanned query on the path, with nothing for the gates or the reviewer to compare it against.

Retries and failure recovery#

Retries are typed. A parse failure or a soft gate rejection goes back to the model with the failed check named, and code caps the attempts per step. A hard fail is not retried, because rewording a DROP statement does not make it safe. A warehouse timeout is retried once with a narrower query if the plan allows one, and otherwise the step fails and the orchestrator decides what to do next.

A failed step does not have to fail the whole question. The orchestrator can return a partial answer, labelled as partial and naming the step that failed, or escalate to an analyst with the plan and the completed results attached. Every step is a read and its result is stored, so a resumed run restarts from the last good step without repeating queries that already succeeded.

Tool routing and bounded loops#

Tool routing picks which capability serves each step: the metric layer, a saved analysis, free SQL, or a clarifying question to the user. Jev's catalogue select and needs-a-tool questions make this choice, with a none-of-these option that routes to a person. Routing a definition question to free SQL wastes a query, and routing an open analysis to a saved report returns a confident answer to a different question.

Every loop has limits in code: steps per question, retries per step, bytes per question and wall-clock time. Jev's supervisors can stop a stuck loop early, such as a model rewriting the same rejected join with small variations. The limits guarantee the loop stops even when a supervisor misreads it, and an exhausted limit ends in a partial answer or an escalation, never a guess.

How do you make every answer auditable?

Record one trace per question that links every step, so anyone can follow an answer from the question to the number. When a finance lead challenges a figure, the useful response is the plan, the SQL, the gate decisions and the rows behind it. A rerun is a weaker response, because the data or the definitions may have changed since. The table lists what each step of the trace holds.

Every gate writes a decision record even when it approves. Approvals are what let you measure false blocks and find checks that never fire, and a log of denials alone cannot tell you whether the gates are too loose. Our guide on building an audit trail for AI decisions covers connecting these records to the business case each answer served.

Reproducible replays#

A replay reruns a past question with everything pinned: the model version, the prompt and Jev question versions, the metric definitions, the allowlist, the thresholds and the schema. Data changes too, so we pin it where the warehouse allows. Snowflake Time Travel and BigQuery's FOR SYSTEM_TIME AS OF clause can query a table as it stood at a past time, within each warehouse's retention window.

A replay that differs from the original then has a cause you can name, such as a new definition, a new model, or data that changed. Replays also test changes. Before a new prompt or model goes live, we replay a sample of past questions against it and compare its answers and gate decisions with the originals, which catches regressions the fixed eval set does not cover.

Metric-definition change history#

Metric definitions live in version control with an owner, and every answer cites the version it used. When a definition changes, the history shows who changed it and why, and the traces show which past answers used the old version. A changed number becomes explainable, and saved analyses built on a definition that no longer holds can be found and flagged instead of drifting quietly.

How do you govern an analytics agent?

Govern it with the controls you already apply to people with warehouse access, plus change control over the parts only an agent has. Access follows the asking user. Everything that shapes an answer, such as metric definitions, prompts, tool descriptions, allowlists and thresholds, changes through review and an eval gate, never at runtime. The table sets out who approves each kind of change and what has to pass first.

Data access and role-scoped warehouses#

Queries run with the asking user's access, never through a shared service identity that sees everything. In practice that means a role per user group granting the reviewed views, row access policies for tenants and regions, and separate compute per role group, so a heavy exploratory user cannot slow finance during the close. Our guide on scoping AI agent permissions covers binding each action to a principal and a task.

Allowlists have owners too. A table joins the allowlist when its data owner has reviewed it, its columns carry sensitivity tags, and its join keys are declared for the parser. A table added to unblock one question, without that review, becomes a path around the masking and row policies for every later question.

Personal data handling#

Personal data is handled at four points, and each one assumes the others can miss. Columns are tagged in the catalogue, and masked views hide them from roles that do not need them. The parser rejects tagged columns outside the allowlist, and result shaping redacts any that still reach results. Logs hold row summaries instead of raw rows, with the same retention policy as the data they describe.

When the catalogue has no tags yet, or when names like notes and col_12 hide what is in the values, a separate discovery job can propose them. It runs a read-only sample query with a hard row limit per column, then classifies each column from the name, the type and the sample values. Pattern checks catch emails, phones and similar shapes. A small language model helps when the name is opaque or the values are messy free text. Suggestions stay suggestions until a data owner accepts them into the catalogue; only then do the allowlist, the masked views, the parser and result redaction treat the column as personal data. The samples themselves are personal data, so the job does not log raw values, does not run on the question path, and does not send unbounded extracts to a frontier model when a small local classifier will do.

The logging point is the one teams miss. A trace that stores raw rows turns the audit system into a second copy of the data it protects, readable by everyone who can read logs. Storing the shape, ranges and counts keeps the trace useful for review without widening access to what it describes.

Eval gates before promotion#

No change reaches users without passing the eval set in CI, and a drop in business-truth accuracy or a rise in false blocks stops the promotion. The gate runs on the question classes the change touches, so a new revenue definition runs the revenue questions and a model upgrade runs everything. The evals section below lists what the set measures, and our guide on how to build LLM evals covers building it.

Where should a person be in the loop?

Put a person where their judgment changes the outcome, and keep them off the routine paths the gates already cover. An analyst approving every safe SELECT against a governed metric adds delay and learns to click approve without reading. An analyst reviewing a first-time metric ambiguity or a high-cost query adds knowledge the system does not have. The matrix sets out the triggers we route to a person, and the path we keep people off.

Every escalation arrives with its evidence from the trace, and the reviewer's decision goes back into it. A reviewer who sees only the final answer can approve or reject it but cannot say which step was wrong, so the system learns nothing. Reviewed escalations become eval cases, and a trigger that reviewers approve almost every time is a candidate to move to allow with audit.

How do you evaluate an analytics agent?

Evaluate the answer a business owner would accept, and the behaviour of the gates, rather than whether the SQL matches a reference query. SQL exact match fails in both directions. Two different queries can both be correct, and a query that matches the reference can still be wrong if the reference encodes an outdated definition.

Execution accuracy, which compares result sets, is better. Public text-to-SQL benchmarks such as Spider and BIRD report it, and their scores say little about your schema, your definitions, or your gates. We read them as background on the model, and build our own set for the agent.

We build the eval set from real questions, taken from the shadow stage and from the requests analysts already get. Each case carries the answer an analyst who owns the metric would give, with the tolerance they accept, and cases where the right behaviour is to abstain or ask. Our guide on how to build LLM evals covers building a set from real failures and keeping it from going stale. The table lists what we measure.

Gate metrics need a denominator of reviewed cases, never only counts. A falling block rate can mean the model writes better SQL, or it can mean a parser change stopped detecting a pattern. Sampling blocked and allowed queries for review, on a schedule, separates the two. The same review feeds the threshold changes, which go through code review like any other change, and the observability stack should let anyone follow one question through its plan, its SQL and its answer.

The escaped-wrong-answer count matters most of all these. Every escape is a failure mode none of the gates knew about. Tracing it to the gate that should have caught it, and adding that check, is how the stack improves over time.

How do you release an analytics agent?

Release it one question class at a time, through three stages, and let evidence move each class forward. A question class is a group such as revenue by region from the metric layer, or ad hoc funnel questions needing free SQL. Classes differ in risk, so they should not share one switch. The review matrix above applies within each stage, and the stage decides how much of it runs. Our guide on rolling out AI agent autonomy in five levels sets out the general model, and the stages below apply it to analytics.

This figure is an illustrative model of the release order we use. It reports no measured result, and the pace depends on traffic and risk.

The matrix sets out what changes at each stage, what has to be true to move on, and what sends a class back.

Shadow comes first because it costs nothing in trust. Analysts keep answering questions as they do today, the agent answers the same questions alongside, and every disagreement becomes an eval case. Skipping it means the first measure of the agent's accuracy is a business user acting on a wrong number.

A class going back a stage is routine. Schemas change, a new source table arrives, and a metric definition gets revised. Each can break a class that was working. Watching gate-trip rates per class tells you which class moved, so you demote that one and leave the rest running.

Common questions

What is an analytics agent?#

An analytics agent answers business questions by planning, writing and running queries against a data warehouse and describing the results. In production it is a layered system: intake and intent, a governed metric layer, planning, SQL generation, validation gates, execution, result shaping and answer synthesis. Memory and a tool boundary serve every layer, and a trace records every step.

Is it safe to let an AI agent run SQL against a production warehouse?#

It can be, if the safety does not depend on the model. The agent connects with a read-only role limited to reviewed views, and every query is parsed and rejected unless it is one allowlisted SELECT. An EXPLAIN or dry-run check blocks large scans, and the warehouse enforces a timeout and a byte cap. Each layer assumes the one above it can fail.

Do we need a semantic layer before building an analytics agent?#

For questions about core metrics, yes, or something that does the same job. Without governed definitions the agent invents one in each prompt, and two answers about revenue can disagree. An existing semantic layer is a good source. Questions outside it can still use free SQL, at a lower level of autonomy and with the same gates.

Why is a read-only database role not enough?#

A read-only role stops writes, and most analytics failures are not writes. A read-only query can scan a full fact table, repeat rows through a bad join, return personal data, or produce a number built on the wrong definition. The parser, the EXPLAIN gate, the metric layer and the answer filter catch those, and the role remains the backstop.

Does an EXPLAIN gate catch every expensive query?#

No. EXPLAIN returns estimates, and stale statistics or late partition pruning can make a costly query look small. The gate stops most runaway queries early at little cost, and the statement timeout and maximum bytes setting in the warehouse catch the rest. In PostgreSQL, never use EXPLAIN ANALYZE in the gate, because it executes the query.

Where does Jev fit in an analytics agent?#

Jev judges the decision edges of the pipeline and never runs or edits SQL. Hard failures, such as a write statement or a scan over the hard budget, are denied in code without asking Jev. Soft failures and warning-band results go to Jev as typed questions, and code turns its calibrated answer into deny, escalate to an analyst, or allow with an audit record.

How do we measure whether an analytics agent is accurate?#

Compare its answers with the number an analyst who owns the metric would give, on real questions, within the tolerance they accept. Track definition fidelity, abstain behaviour, trip rates for each gate, false blocks, and wrong answers that passed every gate. SQL exact match is a poor measure, because different queries can be correct and a matching query can use an outdated definition.

Should an analytics agent be a workflow or an autonomous agent loop?#

Mostly a workflow. The outer shape of intake, plan, gated steps and synthesis is fixed code, so the same question runs the same path and returns the same number. The model has freedom inside each step, where it writes the plan and the SQL. A free loop fits open exploration with an analyst watching, and even then every query runs through the same gates and limits.

What should an analytics agent log?#

One trace per question that links the question, the plan, the SQL as executed, the EXPLAIN estimates, every gate decision with its reason, a summary of the rows, and the final answer with its citations. Log approvals as well as denials, pin the versions of the model, prompts and metric definitions so answers can be replayed, and keep raw personal data out of the logs.

When should a person review an analytics agent's work?#

When their judgment changes the outcome. That covers a parser soft fail or cost estimate in the warning band, the first time a team's question is ambiguous between metrics, high-blast-radius queries, low-confidence answers and policy trips. Routine SELECT queries on governed metrics that pass every gate should run without review, with a sample checked on a schedule.

When should an analytics agent refuse to answer?#

It should abstain when it cannot ground the answer. That covers a question that matches no governed metric with enough confidence, a query that keeps failing the gates, or a draft whose numbers cannot be traced to the result. It should say what it would need, such as a clearer metric or a narrower time window, rather than send its best guess.

Further reading

• Introducing Jev into agentic workflows. The five-step Jev sequence and the rule this guide applies to analytics.
• Shared business context for every AI agent. Governed metric definitions with owners and versions, served to every agent.
• How to scope AI agent permissions. Binding each query to the asking user and task.
• How to roll out AI agent autonomy in five levels. The release model the three stages apply.
• How to build LLM evals. Building the business-truth set from real failures.
• How to build an audit trail for AI decisions. Where each allow-with-audit record belongs.
• Agent playbooks as MCP tools. Publishing saved analyses as tools other agents can search.
• MCP tool design: fewer, larger tools. Why a tool that runs any SQL is one tool in name and every operation in practice.
• sqlglot, SQL parser and transpiler. The parser used in the sketch, with dialect support for the major warehouses.
• PostgreSQL, EXPLAIN. Plan output formats, and the ANALYZE option that executes the statement.
• Google Cloud, BigQuery dry runs. Estimating bytes processed without running a query.
• Snowflake, EXPLAIN. Partition and byte estimates per operation.
• Snowflake, Time Travel, and Google Cloud, BigQuery historical data. Querying a table as it stood at a past time, for replays.
• Spider, a text-to-SQL benchmark, and BIRD, a text-to-SQL benchmark on large databases. Public execution-accuracy benchmarks, useful as background and not as a measure of your agent.