An AI SRE is an agent that takes the first response to a production alert. It reads the alert, queries logs, metrics, traces and recent deploys, and ranks the likely root causes with the evidence for each. Then it proposes a fix or opens a tested pull request for an engineer to approve. As of October 2026 these agents diagnose well in contained tests, and two of them score 100 percent on the public SREGym-Lite benchmark.
Key facts
Definition
An AI SRE agent investigates a production alert before an engineer does, ranks root-cause hypotheses with the queries behind each, and proposes a fix or a tested pull request for approval.
How Bigcircle does it
- Bigcircle's design puts the agent on every alert as first responder and requires each hypothesis to carry its supporting queries and the check that would disprove it.
- Bigcircle keeps mitigation behind named tools with a rate limit, a cost fuse and a human approval, and opens it one fault class at a time on the agent's record against replayed incidents.
- On Skionis, Bigcircle's DevOps agent for a live AWS account, the agent reads live account state before it plans and an engineer approves a diagram rather than template code.
Comparison
SREGym-Lite tests an agent on 17 clean faults, while a replay set built from your own postmortems tests it on undocumented services, sampled-away metrics and dependencies another team owns.
Key number
Daimler Truck's self-healing SRE system on AWS cut resolution from 2 to 3 hours per incident to about 15 minutes, including human review.
That score does not decide how much to let one touch in production: the harness around the agent decides it, together with the agent's record on your own incidents. Getting both right is harder than picking a vendor, because the benchmark faults are clean and your incidents are not.
The design we recommend comes from Skionis, a DevOps agent we built that applies changes to a live AWS account. The engineers who would run it refused to let an agent near production, and they changed their minds because of the review surface, not the model: every step became one a machine checks or a person approves.
An AI SRE agent meets the same refusal and gets past it the same way. It answers every alert as first responder, ranks its hypotheses by the evidence behind them, reaches production only through gated tools, and earns more autonomy from its score on your own replayed incidents.
This guide starts with what the agent does on-call and what this month's launches show, then covers how root cause analysis works and fails, why a benchmark is not your estate, and what to gate. It ends with how to score AI SRE tools on your incidents, what the harness holds, and when an incident agent is the wrong next step.
What is an AI SRE agent, and what does it do on-call?
An AI SRE agent does the part of on-call that comes before anyone decides anything. When an alert fires, it checks whether the issue is already known and gathers the telemetry an engineer would open first, then forms hypotheses about the cause, tests them against the data and writes up what it found. Depending on its permissions, it then stops, suggests a fix, opens a pull request, or runs a mitigation such as a rollback.
Agentic SRE is the name for running reliability work this way, with agents that query and act on systems. The definition we work from is narrower and more useful to a buyer: an AI SRE is a first responder whose every conclusion comes with the queries that produced it. A confident paragraph with no evidence attached saves nothing, because the engineer has to repeat the investigation before acting on it.
Give it the reach an on-call engineer has on the first page of a runbook, and no more. That means read access to logs, metrics, traces, the deploy history, the code and past incident write-ups. Write access comes later and separately, and most of this guide is about when.
What do the October 2026 launches show?
Five vendors shipped or opened AI SRE products between 1 and 21 October 2026. Read the table as a picture of the category rather than a ranking, because each number in it is the vendor's own claim, measured on a test the vendor chose.
| Product | Date | Availability, as of 26 October 2026 | What the vendor claims |
|---|---|---|---|
| Devin Oncall, from Cognition | 15 October | Generally available | 100 percent end-to-end on SREGym-Lite. At AngelList, 3.5 times more correct root causes than its in-house harness, and half of incidents handled end to end |
| Bits Investigation on Scout 1.0, from Datadog | 15 October | Scout in closed preview | Scout 1.0 medium matches two leading general-purpose models at 3 to 8 times lower estimated inference cost, on Datadog's internal AlertEval benchmark |
| Elastic nightshift AI SRE | 8 October | Private preview on Elastic Observability Serverless | Tests several root-cause hypotheses in parallel, with the queries and reasoning visible to the engineer |
| Cortex XCOR Investigations agent, from Palo Alto Networks | 1 October | Introduced with the Cortex XCOR platform | Root cause and remediation in under 3 minutes on average, with a 75 percent success rate on root-cause analysis |
| Triago, from Orchestra | 21 October | Included in all Orchestra tiers | For data-pipeline incidents: groups alerts into incidents, diagnoses them and proposes fixes |
Every product in the table diagnoses first and treats action as a setting the customer controls. Palo Alto Networks makes human review the default, and Datadog lists autonomous remediation as a plan rather than a feature. The numbers also measure different things. By Datadog's own footnote, its scores rate agreement with explanations written by experts on a 0 to 100 scale, and they are not percentages of incidents solved.
Use these claims to build a shortlist and nothing more. A 75 percent root-cause rate and a 3.5 times improvement cannot be set side by side, because neither tells you which incidents were tested or how "correct" was judged. Ask each shortlisted vendor to run on your replayed incidents instead, as the scorecard section below sets out.
Triago answers a question buyers often ask. Orchestra calls it an agent for incident management, and most of its work is the queue around the investigation. In Orchestra's own example it turns 240 near-identical failure messages into 3 threads, and it caps alerts at three per incident by default, raises a ticket and routes the incident to its owner.
AI incident management is that coordination layer, and an AI SRE is the investigation and the fix inside it. Start with coordination if you have neither, because grouping and routing an alert never changes a production system and so carries far less risk than an agent that can act.
How does AI root cause analysis work, and where does it fail?
AI root cause analysis is a loop. The agent forms a hypothesis, queries the telemetry that would confirm or refute it, then updates and repeats until one explanation fits the evidence and the others are ruled out. The products above add memory of past alerts, so a known issue joins its earlier investigation. Some add a second model whose only job is to check the first.
Plan for premature closure. Cognition names this failure in its own launch post: in production, agents "still stop at the first plausible root cause", and an engineer has to tell them to dig deeper. Its answer is a review agent that runs beside the investigator and looks for correlation read as causation, contradictory telemetry dismissed, and a trigger confused with the underlying problem.
Put that check into the output format rather than into another prompt. Require four fields on every hypothesis: the claim, the queries and log lines that support it, the check that would disprove it and whether it ran, and what was ruled out on the way. Rank by evidence, so a hypothesis whose disproving check never ran sits below one that survived its check. A plausible story with no attempted refutation then cannot sit at the top of the list, however fluent it reads.
Ask for trigger and cause as separate fields too. "Errors rose after the last deploy" is a trigger. "The deploy halved the connection pool, which saturated at peak traffic" is a cause, and only the cause says whether a rollback fixes anything. Score the agent on both, because one that reliably finds triggers can look excellent on a benchmark and still send your team to the wrong fix.
A wrong early reading compounds, so the check belongs on every step and not only on the final answer. On Solarpunk, a desktop agent we built, a critic decides after each action whether to continue, retry or escalate. Without it, one bad observation carries through every later step, and the agent reports success on work it never completed. A misread dashboard in an investigation does the same to every query after it.
Does a benchmark score tell you an agent will work on your systems?
It tells you less than the headline suggests, and the benchmark's own numbers show why. SREGym-Lite has 17 faults, each based on a real production incident and injected into a live system. An agent passes a fault only if it names the root cause and mitigates it in the same run.
On the leaderboard as read on 26 October 2026, Cognition's agent running Claude Opus 5.5 scored 100 percent with a time to mitigation of 141 seconds, and CloudThinker running GPT-6 Astra also scored 100 percent. Cognition puts the previous generation of models at 50 to 60 percent on the same test, so the benchmark saturated within a few months.
A saturated benchmark cannot separate the agents at the top. The leaderboard also ran Cognition's agent at 2 attempts per fault and the others at 3, so the rows were not run the same way. Cognition makes the larger point itself: production means legacy systems and searches across huge volumes of telemetry, which no fixed set of 17 faults reproduces.
Treat the benchmark as a floor. An agent that cannot pass SREGym-Lite does not belong on your shortlist, and one that passes can debug an unfamiliar system when the fault is clean. Your incidents are rarely clean, because they involve the service nobody documented, the metric sampled away to save storage, and the dependency another team owns. Only a test built from those incidents shows how the agent does on them.
What should an AI SRE agent be allowed to do on its own?
Grant autonomy in four stages, and move one fault class at a time rather than the agent as a whole. A fault class is a kind of incident with a known shape, such as a pod killed for running out of memory, a bad configuration push, or a certificate close to expiry. An agent can earn the right to roll back a bad deploy long before a database failover.
| Stage | What the agent may do | What has to be in place first |
|---|---|---|
| Investigate | Read telemetry, code and past incidents, and post findings | Read-only credentials, separate from any write path |
| Suggest | Propose a fix or a mitigation in the incident channel | Hypotheses with their evidence, so the on-call can check the reasoning quickly |
| Open a tested pull request | Apply the fix in a sandbox, run the checks, open a pull request for review | An allow-list of repositories, sandbox validation, and a bot identity of its own |
| Mitigate a named fault class | Run one scoped action, such as a rollback, a scale-up or a flag change | A record on replayed incidents of that class, a rate limit, a cost fuse, and a fresh read of live state before acting |
The self-healing SRE system Daimler Truck built with CGI on AWS shows the third stage working. Its SRE agent passes each alert to an investigation agent that is read-only, and a separate coding agent applies the proposed patch only to a repository on its allowed list, and only when the fix is marked applicable. It validates the patch in a sandbox and opens a pull request for an engineer to approve.
Five safeguards sit around that flow: de-duplication of repeated alerts, the repository allow-list, sandbox validation, rate limiting, and a cost alarm that shuts the setup down above a threshold. The AWS write-up puts resolution at about 15 minutes per incident including human review, against 2 to 3 hours before.
Copy its narrowest decision first: the system acts only on error cases the team had already tested thoroughly, and the team widens that set on purpose. Widening the set only after a class has been tested keeps the first failure of a new class in a replay rather than in production.
Promote a fault class to the fourth stage only on evidence you wrote down before the agent started. Record how many replayed incidents of that class it handled, how many it diagnosed correctly, and whether it ever claimed a fix that did not hold. Reset the class to the third stage after any false "resolved" claim.
The reset is strict because the two kinds of error cost different amounts. A wrong hypothesis the agent flagged as uncertain costs the on-call a few minutes, while a confident claim that the incident is over can cost a second outage, and a customer may find it first.
Scope each stage as its own grant, as AI agent permissions describes, so investigation credentials cannot stretch into a write. Design the approval step as carefully as the agent, using human in the loop AI agents for who approves what. The figure below shows the stages and where the gate sits.
Blue stages run on every alert once the table's prerequisites for them are in place. Orange stages wait behind the gate, which opens for one fault class at a time on the agent's record against your replayed incidents. The figure follows one alert, and the four stages in the table above say how far along it the agent may go on its own.
How do you evaluate AI SRE tools on your own incidents?
Build a replay set from incidents you have already resolved, and score every candidate on it, including any agent you build yourself. For each incident, keep the alert, a snapshot of the telemetry from when it fired, the code and deploy history as they stood then, and the root cause your engineers wrote in the postmortem.
Datadog built its internal AlertEval benchmark from the same ingredients: real alerts, archived telemetry and root causes written by engineers. Start with the recent incidents that have a written root cause, and add each new one as it closes. Run every candidate on the same set with the same access, and score it on the measures below.
| Measure | How to score it | Why it matters |
|---|---|---|
| Root cause correct | Matches the postmortem's cause, not only its trigger | A trigger-only answer sends the team to a fix that does not hold |
| Evidence attached | Each claim links the query or log line behind it | An unsupported answer has to be investigated again by hand |
| Time to a useful hypothesis | Time until the on-call has a hypothesis worth acting on | The on-call decides early whether to trust the agent or start over |
| Time to mitigation | On replays where mitigation is in scope | The outage length the business sees |
| False "resolved" claims | Every claim of a fix that did not hold, on replay and live | One of these undoes the trust that many correct diagnoses built |
| Pages avoided | In a live shadow run, alerts closed with no engineer paged and no reopen | Replays have no pager, so this needs live traffic |
| Cost per investigation | Tokens and tool calls per alert, on your own traffic | Running on every alert multiplies it by your alert volume |
Count false "resolved" claims separately, and never average them into an accuracy score. A tool with a higher diagnosis rate and one false all-clear is worse on call than one that diagnoses less and says when it is unsure, because the all-clear stops the humans from looking.
Check every "resolved" against the alert's own signal, the error rate or latency that fired it, back within its threshold, rather than the agent's account of its work. Why AI agents report work as done covers why an agent's report of its own completion is the wrong thing to trust.
Run a live shadow period before any action stage. The agent investigates real alerts and posts findings while the on-call works as usual, and you compare its hypothesis with what the engineer found on each page. Replays tell you about the past, while a shadow run shows the alerts your estate produces this month, including the ones nobody has written up.
Price the investigation on your alert volume, not per seat. Cost per investigation is the stated reason Datadog trained Scout, and grouping known issues is the first saving. Cognition claims its responders skip roughly 60 percent of investigations by recognising earlier work and grouping related alerts. Measure that share on your own alert stream before relying on it, because it depends on how repetitive your alerts are.
What does the harness around the agent need?
On Skionis, adoption came from the review surface and not from the model, and four of its parts carry over to an incident agent almost unchanged. They are the work behind our Cloud and DevOps and AI observability services.
Read live state again before acting. Skionis reads the account's limits, quotas, regions and existing resources before it plans, because only the account can say whether a change will apply. Its remaining failures are state failures: a change set that fails partway, a resource that changed after the read, a quota that moved between plan and apply.
An incident moves faster than a cloud change: other responders act, autoscaling fires and a second deploy goes out, so a rollback proposed early can be wrong a few minutes later. Make every mitigation tool read the state it depends on again, and refuse when that state has changed since the hypothesis was formed.
Give the agent named tools and no shell. Skionis reaches AWS only through purpose-built tools, with no general shell and no way for a plan step to become an arbitrary API call. Give an AI SRE agent one tool per mitigation, with its scope written in code, such as rolling back this service to its previous release, or scaling this deployment within set bounds. A fix-writing agent also runs code it wrote, so it needs the limits in AI agent sandboxing as well as its tools.
Design a review surface the on-call will read. On Skionis the engineer approves a diagram of what will exist rather than template code, because engineers do not read raw template diffs carefully at the tenth change of the week. The incident version is one screen with the top hypothesis, its evidence, the action, the signal that should recover and the rollback. An approval request that arrives as a page of logs in the middle of the night tends to be approved unread, and the gate then records a name and checks nothing.
Keep one record per incident, alert through outcome. Skionis keeps one record linking the request, the account facts, the generated template, every validation failure, the architecture preview and the resources that now exist. Its validation failures are the system's own log of what the model gets wrong. Template errors before deploy fell from around 10 percent to 4 percent once that loop was in place, by our own measurement.
For an incident agent, the record holds the alert, each hypothesis with its queries, the causes ruled out, the approver, the action and the recovery signal. Turn every hypothesis the on-call rejected into a new replay case, and keep the record as an AI audit trail does.
Watch the agent as a production system in its own right. It calls models and tools, it can fail quietly, and its quality can drift while every request returns a success status. AI agent monitoring covers the two layers to watch and who owns each.
When is an incident agent the wrong next step?
Write postmortems first if your incidents have no written root cause. Without them there is nothing to score the agent against, and you will be choosing on vendor claims. A few months of recorded causes is enough to start a replay set, and the habit improves your human on-call whatever you buy.
Fix alert routing first if alerts arrive in duplicate and with no owner. An agent that investigates every duplicate spends money on noise. A coordination layer that groups and routes alerts carries less risk and makes the later evaluation possible.
Check what telemetry you keep. An agent cannot find a cause in data that was dropped to save storage, a point Elastic makes in its own launch. Look at sampling and retention on the services that page most before you score anyone.
If your estate is small and incidents are rare, the replay set will stay too small to earn the fourth stage. Keep the agent at investigate and suggest in that case, where it still saves the first stretch of each page, with no write path to production.
The strong case is the reverse: a steady volume of alerts, fault classes that recur, and postmortems that name causes. Put the agent on every alert from the start, and let its record on your incidents decide how far it goes.
Common questions
What is an AI SRE?#
An AI SRE is an agent that takes the first response to a production alert. It checks whether the issue is already known and queries logs, metrics, traces and recent deploys. It ranks root-cause hypotheses with the evidence for each. Then it suggests a fix, opens a tested pull request, or, where allowed, runs a scoped mitigation such as a rollback.
What is agentic SRE?#
Agentic SRE is reliability work run by agents that query and act on systems, rather than by dashboards that wait for a person. In practice an agent investigates each alert first and hands the on-call engineer ranked hypotheses with evidence. Any action it takes in production goes through scoped tools and an approval step.
How does AI root cause analysis work?#
The agent forms a hypothesis, queries the telemetry that would confirm or refute it, updates and repeats until one explanation fits the evidence. Its common failure is stopping at the first plausible cause. Require each hypothesis to carry its supporting queries and a disproving check, and score trigger and cause separately.
Is AI incident management the same as an AI SRE?#
No. AI incident management is the coordination layer: grouping duplicate alerts into incidents, routing them to owners, raising tickets and posting summaries. An AI SRE is the investigation and fix inside an incident. Coordination never changes a production system, so it is the lower-risk place to start if you have neither.
How do you evaluate AI SRE tools?#
Replay incidents you have already resolved, with the telemetry and code as they stood, and score each tool against the root cause in your postmortems. Measure root cause correct, evidence attached, time to mitigation and cost per investigation. Count false resolved claims separately. Then run a live shadow period to count pages avoided.
Should an AI SRE agent be allowed to fix production on its own?#
Only for a named fault class, after a record on replayed incidents of that class with no false resolved claims. Until then keep it investigating, suggesting and opening tested pull requests for review. Give each mitigation its own scoped tool, a rate limit and a cost fuse, and have it read live state again before it acts.
Further reading
- Cognition, Introducing Devin Oncall, 15 October 2026. General availability, the SREGym-Lite result, the AngelList example, the review agent, and the share of investigations its responders skip.
- SREGym leaderboard, read 26 October 2026. The 17-fault SREGym-Lite results, attempts per fault, and times to mitigation.
- Datadog, Introducing Scout 1.0, 15 October 2026. Closed preview, the 3 to 8 times estimated cost figure on AlertEval, and the footnote on what its scores measure.
- Elastic, Elastic nightshift AI SRE, 8 October 2026. Private preview, parallel hypotheses, visible reasoning, and telemetry dropped to control cost.
- Palo Alto Networks, Introducing XCOR Operator and AI SRE Agent, 1 October 2026. The under 3 minutes and 75 percent claims, and human review as the default.
- AWS, Automating incident resolution with a multi-agent self-healing SRE solution on AWS, Eeran Maiti, Simon Rapp and Wael Dimassi, 12 October 2026. The Daimler Truck system, its five safeguards and the 2 to 3 hours to 15 minutes figure.
- Orchestra, Announcing Triago, 21 October 2026. Incident grouping, alert budgets and the 240 messages to 3 threads example.
- Treating the agent as untrusted on live AWS, the Skionis build, and the Solarpunk build for the critic step.
- AI agent monitoring, AI agent permissions, human in the loop AI agents, AI agent sandboxing, AI audit trails and how to build an LLM evaluation.