To move an AI pilot to production, run it on real inputs before changing anything and sort what fails. A pilot that does its core task and fails at the edges can be hardened: an eval set first, then checks in code before an agent acts, a locked-down data path, cost and latency under real traffic, and a safety net. A pilot that fails at the core task needs a rebuild against a written acceptance test, because hardening it only makes the failure better logged.
Key facts
Definition
Moving an AI pilot to production starts with sorting its failures on real inputs, because a pilot that fails at the edges can be hardened and one that fails at the core task needs a rebuild.
How Bigcircle does it
- Bigcircle builds the eval set from real inputs before changing anything in a pilot, so each later change can be measured against it.
- On Bigcircle's DevOps agent for live AWS accounts, template errors before deploy fell from around 10% to 4% once a validation loop checked every template, by Bigcircle's own measurement.
- Bigcircle keeps credentials out of the prompt on its desktop agent, injecting them from an encrypted local vault into the tool adapter at execution time.
Comparison
Cost per call can look lower on a model that fails and retries, while cost per completed task shows what a finished piece of work costs.
Key number
Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025.
This guide is for a team that already has a pilot working in a demo. If the question is why pilots stall before they get that far, why AI pilots fail argues the causes and the decisions to settle before budget goes in. Here the order matters more than any single step, because a change made before the eval set exists is a guess nobody can check.
| Criterion | Harden the pilot | Rebuild against a written test | Settle the value first |
|---|---|---|---|
| When it fits | The pilot does its core task on real inputs and fails at the edges | Most failures on real inputs sit in the core behaviour | Nobody can say what the system is worth once it works |
| What you do first | Build the eval set from real inputs, before changing the system | Write what done means as a test, then scope the build against it | Write down the metric, its baseline and the date of the verdict |
| What it cannot fix | A retrieval step that never finds the right passage, or a task the model cannot do | A definition of done that is still a feeling | Anything in the engineering, since value is a business question |
| How Bigcircle handles this | Production hardening: we define what good means and build the eval set before changing the system, then add model routing, a locked-down data path and a safety net | A fixed-scope build: discovery, build, evals and deployment against a scope and price agreed before work starts | Neither engagement answers this question, so settle it before you buy either |
Some things are the same whichever route you take with us. You own the code and the IP, and the work happens in your repositories. In each one the agent is treated as untrusted and checked in code before it acts.
How do you know if an AI pilot is ready for production?
A pilot is ready for hardening when it already does its core task on real inputs and fails around the edges. Edges here means the inputs nobody tried, cost and latency under real traffic, personal data on its way to a model provider, and an agent acting with no check in code. Hardening fixes those. It does not fix a retrieval step that never finds the right passage, or a model that cannot do the task at all.
You can find out which kind of pilot you have before talking to anyone. Run it on a few dozen real inputs rather than the demo set, read every failure, and sort them. The guide to LLM evaluation covers that reading in detail. If most failures sit in the core behaviour, the work ahead is a rebuild, and calling it hardening will only stretch the timeline.
The demo set is the wrong test because it was chosen by the people who built the pilot, usually while they were building it. Every input in it is one the system was tuned on. Real inputs bring the misspelled query, the scanned page, the request that needs two tools, and the question the documents do not answer. A pilot that handles those badly but gets the plain cases right is a hardening job. One that misreads the plain cases is not.
How do you move AI from pilot to production?
Harden in this order. Each step is measured on the eval set built in the first one, so you can see what each change bought, and a later step cannot hide a regression from an earlier one.
| Step | What it settles | What happens without it |
|---|---|---|
| Eval set from real inputs | What good means, as cases with an expert's verdict | Every fix after it is an opinion, and a change that helps one input can break three others |
| Edge cases and refusals | What the system does on inputs nobody demoed, including when it should decline | The first real users find them |
| Checks in code before an agent acts | Which actions can run, with what arguments, and who approves them | The prompt is the only control |
| A locked-down data path | What leaves your environment, and where credentials live | Personal data and keys travel in the prompt |
| Cost and latency under real traffic | Cost per completed task, and model routing where the evals allow it | The bill and the wait grow with usage in a way the demo never showed |
| A safety net | Monitoring, bounded retries, escalation and rollback | Failures reach you through customers |
| Acceptance written as a test | The bar for go-live, and who runs it after handover | Acceptance becomes an argument about the demo |
The eval set comes first because nothing else can be measured without it. On our trademark drafting platform, every draft is scored by a legal-integrity evaluation, 0 to 10 across six axes, before an attorney sees it, and the attorney still edits and signs.
We treat the attorney's corrections as the most valuable signal in that system, because they measure the gap automated scoring cannot see. A pilot's eval set starts smaller than that, from the failures you sorted, and it grows each time a new one turns up.
Edge cases and refusals come next, because they are where real traffic differs from the demo. Add the inputs the pilot failed on to the eval set, and add the questions it should decline. A system that answers everything sounds confident in a demo, and in production it is wrong on the questions its documents do not cover.
What has to be in place before an agent acts in production?
An agent needs checks in code that the model cannot talk its way past. On the DevOps agent we built for live AWS accounts, there is no general shell: the agent reaches AWS only through purpose-built tools. Every generated template goes through a validation loop before AWS is asked to start anything, and an engineer approves an architecture diagram of the change before it is applied.
Template errors before deploy fell from around 10% to 4% across the validation loop's introduction. That is an internal measurement.
Rules that can be written down belong in the tool contract rather than the prompt. On the insurance voice agent, built and tested end to end but not yet in production, a payment, claim or policy operation is schema-validated, permission-checked and blocked until any required disclosure has been made. A request that fails any of those checks does not turn into an approximation. It escalates to a person. The guide to AI agent permissions covers how to scope what each tool may do.
The data path needs the same treatment. On Solarpunk, the desktop agent we built for a leader's own systems, credentials sit in an encrypted local vault and are injected into the tool adapter at execution time. They never travel into the prompt.
A pilot that pastes an API key into a system prompt, or sends whole customer records to a model provider to answer one field, has no risk control a security review can sign off. Fix that before any cost work, since routing and caching will send more traffic down the same path.
How do you control cost and latency once real traffic arrives?
Track cost per completed task next to quality from the first day of real traffic, because a pilot's inference bill grows with usage and context in a way the demo never showed. Cost per call misleads, because a call that costs less but fails and retries can cost more per finished task than one that succeeds first time. Route requests to a smaller model only where the eval set shows the quality holds, and keep the eval run that shows it.
Repeated work is often the largest saving. On Prospex, our own prospect research engine, cache-first execution cut repeated external API calls by an estimated 50 to 60 percent on repeated runs. The guides to LLM inference and AI agent cost cover the levers in more depth.
Then put a safety net under the system before it takes traffic. On Solarpunk, retries are bounded, because an agent that retries indefinitely against a broken vendor burns cost and time while looking busy. Work it cannot finish comes back as a flagged exception with the unresolved step and its trace attached, rather than a confident, false "done". The guide to AI agent monitoring covers the alarms and the sampled quality scores that sit on top.
What should the acceptance test say?
Write the definition of done as a test, because it is the contract and a feeling cannot be checked at handover. For a retrieval system, done means the right passage comes back on your questions and every citation resolves. For an agent, done means an independent check that the action happened, separate from the model's own account of it. For anything a customer sees, done also covers what the system must refuse.
A scope can name a measured bar, as the trademark harness does, and acceptance then becomes a reading of the score rather than an argument about the demo.
Acceptance is not the end of the work, and the scope has to say so. A retrieval system keeps meeting questions and documents that nobody tested, so it can only be fully checked while it runs. We deliver the eval set into your repository with the code, and the acceptance test should name who runs it against real questions after handover.
What can production hardening not fix?
Two things sit outside hardening: a pilot nobody can put a value on, and a core that cannot do the task. Check for both before scoping the work. Gartner's list of reasons for abandoning a project after proof of concept is a useful map of where that line falls. In July 2024 it predicted that at least 30% of generative AI projects would be abandoned by the end of 2025, for poor data quality, inadequate risk controls, escalating costs or unclear business value.
Two of those reasons fall inside hardening, and one of them is partly inside. Unclear business value is outside it. If nobody can say what the pilot is worth once it works, a hardened version will be reliable and still unused, so settle that question before you buy the engineering.
A pilot that fails at the core task is outside it too. Hardening a system that cannot do its job adds monitoring to a failure, and the next step is a rebuild with a new acceptance test.
Who should do the hardening?
Your own team, if it can build the eval set and the monitoring as well as call a model API. The guide comparing an in-house AI team, an agency and an AI implementation partner covers that choice. If you bring the work to us, start from what you already have.
A pilot that works in the demo and stalls on real inputs is production hardening. Your team built it, we finish it, and the eval set comes before any change to the system.
A clear problem with no system yet, where you can say what done means, is a fixed-scope build: discovery, build, evals and deployment under one point of accountability, at a scope and price agreed before work starts. A roadmap of AI work with nobody senior to make the hard calls is a studio-led programme, a team of 3 to 8 engineers and QA led by our founder as your fractional CTO.
The public build closest to that last shape is Solarpunk, which our team delivered with a fractional CTO leading the build. Plans completed with no human intervention rose from roughly 55% to 80% once the agent was grounded in the user's own workspace, by internal measurement.
The hard calls on it were architectural. The agent reaches more than a hundred tools by looking up the right one at call time instead of holding all of them in its context, because tool choice degrades past roughly thirty tools.
Bigcircle does not place engineers in client teams. If you already have a senior AI lead and only need more hands, a fixed-scope build against your lead's plan, or a hire, fits better than a programme.
Common questions
How do you move AI from pilot to production?#
Run the pilot on real inputs and sort the failures, then build an eval set from them before changing anything. Add edge cases and refusals, put checks in code before an agent acts, lock down the data path, and measure cost per completed task under real traffic. Put monitoring, bounded retries and escalation under it, and write acceptance as a test someone runs after handover.
How do you know if an AI pilot is ready for production?#
Run it on a few dozen real inputs rather than the demo set and read every failure. If it does its core task and fails at the edges, such as unusual inputs, cost, latency or an agent acting with no check in code, it is ready for hardening. If most failures sit in the core behaviour, it needs a rebuild against a written acceptance test first.
What does production hardening include?#
It starts with defining what good means and building an eval set on real inputs, before anything in the system changes. It then adds model routing for cost and latency, a locked-down data path, checks in code before an agent acts, and a safety net under the system before it takes traffic. Each change is measured on the eval set, so you can see what it bought.
What should an AI acceptance test say?#
What done means, in a form someone can run. For a retrieval system, the right passage comes back on your questions and every citation resolves. For an agent, an independent check confirms the action happened, separate from what the model reports. For anything customers see, it also lists what the system must refuse, and it names who runs the test after handover.
When is a rebuild better than hardening the pilot?#
When most failures on real inputs sit in the core task: retrieval that never finds the right passage, or a model that cannot do the job. Hardening adds evals, checks, cost control and monitoring around a system that works, and it cannot make a broken core work. Scope the rebuild against a new acceptance test written before the build starts.
Do you place engineers in our team?#
No. Bigcircle does not do staff augmentation. Every engagement delivers a system, and the people who build it are Bigcircle's own team, hired and trained by the founder, kept on one product and never subcontracted. If you need permanent AI capability inside your own team, the build-or-partner guide covers when to hire.
Who owns the code on each engagement?#
You do, on all three. The work happens in your repositories under your IP and confidentiality terms, and the eval set is delivered there with the code. On a fixed-scope build Bigcircle owns the delivery of the outcome, not the repository. On a studio-led programme Bigcircle makes the technical calls, and you still own the code and the product decisions.
Further reading
- Why AI pilots fail. Why pilots stall, and the value and ownership questions to settle before you buy the engineering.
- LLM evaluation. How to read your pilot's failures, and how to write the acceptance test.
- AI agent monitoring. The safety net once the system takes traffic.
- AI agent permissions. Scoping what each tool may do before an agent acts.
- AI implementation partner. Whether to build the capability yourself.
- Production AI engineering. The production hardening engagement.
- Case studies named here: trademark drafting, the DevOps agent, Solarpunk.
- Gartner, press release of 29 July 2024. The prediction and its four reasons for abandonment after proof of concept.