The product had to finish cross-system work, not describe it. That single requirement is what separates this from an assistant, and it forces three things a chat product never has to solve: acting inside systems that only expose a login screen, holding a plan together across a hundred-plus tools, and proving afterwards that the work was actually done.
This is the engineering half of the case study. The business read (what changed for the leadership teams using it, and what it was worth) is a separate document.
The problem
The naive architecture is to wire a model directly to each service and describe every tool in the system prompt. It works, convincingly, up to roughly the point where you have thirty tools. Then three separate things break at once.
Tool selection degrades. With a hundred-plus tool definitions competing for attention in one context, the model starts choosing plausible-but-wrong tools, and the failure is quiet: it returns a confident result from the wrong system. The context cost is the visible symptom; the selection error is the expensive one.
Maintenance degrades. Every integration wired directly is its own credential handling, its own error shapes, its own prompt fragments. A vendor changes a response and the break surfaces somewhere unrelated, days later.
And the coverage ceiling is hard. A large share of executive work lives behind SSO and MFA in apps with no usable API. An agent restricted to services with clean APIs cannot finish a routine; it can only finish the part of a routine that happens to be convenient, which leaves a human holding the sequence anyway.
-
Tool selection has to stay reliable as the registry grows past a hundred
-
Credentials must never enter model context, while still being usable inside login-only apps
-
A long plan has to survive a mid-sequence failure rather than derailing on the first surprise
-
Spreadsheet reasoning must be reproducible, which rules out having a model describe a table
Architecture
The system is a desktop runtime rather than a hosted service, and that is the decision everything else follows from. Credentials, the tool registry, retrieval and execution all sit on the user's machine. The model is called as a reasoning service; it is never the thing holding the keys.
Between the model and every integration sits one uniform tool layer. Nothing is wired point-to-point. A tool is a typed contract (declared inputs, declared outputs, declared side effects) and the agent's entire capability surface is the set of registered contracts. That boundary is what makes the action space enumerable, which in turn is what makes governance a property of the system rather than a promise in a prompt.
The credential vault is encrypted and local. Secrets are injected into the tool adapter at execution time, at the point of use, and never travel into the prompt. This is what makes browser-based action inside an authenticated app defensible rather than reckless: the agent drives a session, it does not learn a password.
Design decision: the model is not given the registry. It queries it. Retrieval-before-invocation is the choice that let the tool count grow past a hundred without selection accuracy falling apart.
We shipped that pattern into production before the major labs published it as a documented standard. We are not claiming to have invented an idea, retrieval over a large action space is not exotic. We are saying that when we hit the wall, there was nothing to copy, and the design we arrived at independently is the one the ecosystem converged on months later. That is the useful signal for anyone evaluating who to hand an unsolved problem to.
The business read: what this bought the leadership teams using itHow it works
Plan#
A goal is decomposed into an ordered sequence of steps. The planner is working against the shape of the task, not against a list of a hundred tools, because at this stage it has not been handed any.
Discover#
For a given step, the agent searches the registry and retrieves only the applicable tool definitions before invoking anything. Two effects, and the second matters more than the first: the working context stays small, and the model is choosing among a handful of relevant contracts rather than a hundred competing ones.
Act and observe#
Each step runs as a think-act-observe loop. The tool call executes through the adapter (with credentials injected there and only there) and the raw observation comes back into the loop rather than being summarised away.
Critique#
A critic evaluates the observation against what the step was actually supposed to achieve, and decides: continue, retry, or escalate. This is the component that makes long plans viable. Without it, one bad observation propagates through every subsequent step, and the agent reports success on work it never completed.
Execute code for analysis#
Spreadsheet and CSV work does not go through summarisation. The agent infers the schema, writes code, runs it, inspects the actual result, and refines. The artifact handed back is a notebook that re-runs, which means the analysis is auditable rather than assertable.
Remember#
Local retrieval plus planner memory ground the agent in the user's own workspace and carry context between runs, so a plan for a recurring routine is informed by how that routine went last time.
Control planes
Governance#
The registered tool layer is the action boundary, not a policy applied over a general capability, but the capability itself. Tool contracts bound inputs and outputs. Credential custody sits outside model context. Where the agent cannot proceed within those bounds it escalates; there is no path by which it invents access it was not granted.
Orchestration#
Goal decomposition, then per-step retrieval, then think-act-observe, then critique, then the next step. That loop coordinates the tool registry, authenticated browser sessions, code execution, retrieval and memory, and it holds state across email, calendar, documents, CRM and ERP for the length of a routine.
Observability#
The unit of observation is the whole run, not the individual model call. One trace links goal, plan, the tools discovered, the calls made, the observations returned, each critic decision, every retry, any escalation, the final result, and the resulting state in the target application.
That last element is the one that is usually missing and matters most. Comparing the agent's claimed completion against the actual application state is what catches false completion, an agent reporting success on work that did not land.
When it fails
The failure modes worth designing for are not model failures. They are: a login that does not go through, a page whose state has moved since the plan was made, an action the target system does not actually support, a tool that returns a partial result, and a completion check that does not pass.
Each of those enters a bounded retry or escalation path rather than a free-running loop. Bounded is the operative word, an agent that retries indefinitely against a broken vendor is worse than one that stops, because it burns cost and time while looking busy.
Escalation hands back the specific unresolved step with its trace attached, so the human resolves one thing instead of auditing the whole run. And because the trace distinguishes a planning defect from a provider outage, an expired session, stale state, or an unsupported action, a failure report points at a fix rather than starting an investigation.
Results
The architecture moved leaders from manual coordination to review, and it did so while keeping both the access surface and the failure behaviour inspectable.
Retrieval-before-invocation is what carried the tool count past a hundred; the critic loop is what carried plan length past the first surprise; the local vault is what carried coverage into the systems that only have a login screen. Each of the numbers below traces to one of those three.
-
No-intervention plan completion rose from ~55% to 80% once local retrieval and planner memory grounded the agent in the user's workspace
-
100+ tools registered with reliable selection, via retrieval before invocation rather than a fully-loaded tool context
-
Spreadsheet analysis from hours to under fifteen minutes, as re-runnable notebooks rather than model-asserted numbers
-
Vendor changes contained to a single integration, because nothing is wired point-to-point
Evidence
Directly demonstrable in the system: the tool registry and its size, retrieval before invocation, the encrypted local credential vault, code-execution analysis, authenticated browser action, and the critic loop.
Internal measurements not yet re-confirmed with the client: the 55% to 80% completion change, the 40–60% recurring-work reduction, and the hours-to-fifteen-minutes analysis result. These were measured during the build, on our own instrumentation. We have not restated them as client-verified production figures.
On the timing claim: retrieval-before-invocation went into this product before the major labs published the equivalent pattern as a standard. That is a statement about our own delivery record, and it is datable from the build history rather than from anyone else's announcement.
If you are evaluating this pattern for your own system, the question we would ask first is not which model you are using. It is whether you can enumerate your agent's action surface, and whether you can compare what it claims it did against what actually changed. Everything above is downstream of those two.
How we classify each of these numbers, and the value model behind them