Why AI pilots fail in the enterprise

AI pilots fail in the enterprise mostly for reasons settled before anyone writes code. The wrong problem gets picked, nobody can say what a correct output looks like, the data is missing, nobody owns the system, or the work around it never changes. The model is rarely the cause. A pilot plan that spends most of its effort on the model is spending it on the part least likely to fail.

Key facts

Definition

AI pilots fail mostly on choices made before the code: the problem, the data that grounds it, the measurement, the owner, and the workflow around it.

How Bigcircle does it

1. Bigcircle builds in the client's repositories, so the code, the eval set and the IP stay with the client when an engagement ends.
2. On Bigcircle's DevOps agent for live AWS accounts, engineers came to let the agent own routine cloud changes because they could approve an architecture diagram of each change before it ran.
3. Every return figure on Bigcircle's case studies is either a formula for the client's own numbers or a worked example.

Comparison

A team that can call an API but cannot yet grade the output should partner first, while a team that can build the evals and monitoring is ready to build.

Key number

MIT NANDA's 95 percent figure counts organizations reporting zero return, and it rests on interviews with people from 52 organizations and a survey of 153 senior leaders.

Our view is that each of those failures has a mechanism, and the mechanism tells you what to build before the pilot starts. This guide argues each one from how the system works. Where one of our own systems was built to stop it, we say what we did and link the case study. The table below comes first, because the build, buy or partner choice decides which of these risks you carry yourself.

What percent of enterprise AI pilots fail?

It depends on what the study counted, and we plan against none of the numbers. The figure every article repeats, that 95 percent of organizations get zero return from generative AI, comes from MIT NANDA's 2025 report. It counts organizations rather than pilots, and its results are self-reported. The evidence is a review of over 300 public initiatives, interviews with people from 52 organizations and a survey of 153 senior leaders. The other rates in circulation count different things again, such as projects abandoned after a proof of concept, or companies that have not yet shown value. Treat any of them as evidence about the field rather than about your pilot, because none of them knows which of your decisions is weak.

Every count points the same way: pilots are easy to start and hard to finish. In our experience that follows from how a demo is judged. A demo runs on cases someone picked, and the question it answers, whether the model can do the task, is almost always yes. A pilot runs on whatever the workflow sends it. It has to answer a harder question: does the work get better or cost less once the system is in it? Answering that needs a baseline, a way to grade an output and a workflow that changes, and most pilots start with none of them in place.

Why do AI pilots fail?

Each of the five failures in the opening answer is decided before the build. Each also gets more expensive the longer a pilot runs without it settled, because the pilot keeps producing results that nobody can read. The table sets out what each one looks like from inside a pilot, and what we built on our own systems so that it could not decide the outcome.

Start with the wrong-problem row, because it decides whether the others matter. Every model has an error rate, so the question to settle is how the work around the system handles an error. On the trademark system a draft that is right 95 percent of the time still has to be checked in full by the attorney, which is the work the firm was trying to avoid. On our prospect research engine, a profile that is right 90 percent of the time is not 90 percent useful. One invented detail that a rep repeats on a sales call costs more than the profile was worth.

On the executive operations agent, a routine that is 70 percent finished still has a person holding the sequence, so it has saved almost nothing. None of those are measured rates. They are the arguments that set each design, and a pilot that skips this argument ends up quoting an accuracy figure that nobody can turn into a saving. Engineers who have worked on failed projects say much the same. In RAND's 2024 interviews with 65 data scientists and engineers, 84 percent named a leadership-driven cause as the main reason projects fail.

Trust decides the last row. On the DevOps agent we built for live AWS accounts, the engineers would not let an agent touch production without guardrails they could see. They came to let it own routine cloud changes because they could approve a diagram of each change before it ran. The alternative was raw template diffs, which engineers do not read carefully at the tenth change of the week. Nothing about the model made that happen; the review surface did.

What has to be ready before a roadmap?

Run a readiness check before writing the list of use cases, because the check changes the list. We built a free AI readiness assessment around six weighted dimensions, with weights taken from the Cisco, BCG, AWS and McKinsey readiness frameworks. Technology and infrastructure carries 20 of 100 points, data readiness 18, talent 18, strategy and value 16, adoption and operating model 16, and governance 12. A low score on any one of them narrows which use cases are safe to try first.

Data readiness for a language model system mostly means the documents it will answer from, not training data, because most enterprise systems ground the model in your documents through RAG. Ask whether those documents are current, deduplicated and carry permissions that retrieval can enforce, and whether there is one version of each policy or four that disagree. A system grounded in contradictory documents answers confidently and inconsistently, and users are right to stop trusting it. On our ESG filings system each new filing is checked against what the knowledge base already holds, so a restatement or a contradiction shows up as a flag rather than as two versions of the truth sitting side by side. The guide to RAG in production covers this failure in detail.

Sometimes the data that grounds the system is not a document at all. On the DevOps agent, a model could write a template that was valid against the schema and still rejected by the account. It could be over a quota, in a region where the service was not enabled, or pointing at a resource that did not exist. The agent reads the live account before it plans, because only the account tells you a change will apply. A readiness check that asked only about documents would not have found it.

The single most useful test is whether a domain expert can look at an output and say whether it is correct within a few minutes. That expert verdict is the ground truth, and without it you cannot build evals, the repeatable tests that tell you whether a change made the system better or worse. Without them you cannot tell a real improvement from a lucky demo. Then check the rest:

• Can you draw the current workflow, with its exceptions and owners?
• Do legal and security know the project exists?
• Can your team build the tests and monitoring, rather than only call an API?
• Is there a named executive with a budget that survives one disappointing quarter?

Which use cases should go first?

Three at most, and the most forgiving of them first. Each use case needs its own eval set, its own integration and its own review design, and none of that gets smaller by running fifty in parallel. A backlog of fifty signals ambition to the board and leaves none of them with the engineering depth it needs to reach production.

Filter the candidates in order. Value comes first: time saved, revenue or risk reduced, in numbers someone in finance would sign, because a value you cannot estimate is a value you cannot later show. Feasibility comes next: does the data exist, does the ground truth exist, and is the task within what current models do reliably? Then the blast radius: what happens when the system is wrong, since every model has some error rate, and who catches the error before it costs anything.

The top left quadrant is where a first wave belongs: drafting that a person reviews, triage that routes rather than decides, summaries that link back to their sources. The same 90 percent accuracy is a liability in the top right quadrant and useful in the top left, purely because of what surrounds it.

Prefer a workflow to an agent for the first wave. In a workflow your code sets the path and calls the model at fixed steps. An AI agent chooses its own next step, which buys flexibility and costs the predictability a first pilot is trying to establish. Even on the DevOps agent, a free-form request becomes a structured spec and a fixed chain of tool calls, because a reviewer can only build judgement about output whose shape is stable.

Save an agent for the second wave, once the first has left you an eval set and a review habit to judge it with. An agent can fail in the path it chose as well as in its answer, so judging one needs traces and a grader for each step. A team that has not yet graded a fixed output is not ready to grade a path. The guide to agentic workflows goes through that decision.

When do you build, buy, or partner?

Decide on what your team can grade, not on what it can build. Building is usually underestimated because the model call is a small part of the system. The rest is retrieval, evals, checks on the output, monitoring and integration, and that is the part most pilots never get to. A team that can call an API and cannot yet tell whether an output is right will ship the call and skip the rest. The pilot then reaches its review with nothing to show but a demo.

For the buy column, ask a vendor for their eval results on your document types, and for a pilot scoped to your data. A vendor who cannot show evals is selling a demo. For anything customer facing, also ask who carries the liability for a wrong output, in the contract.

For the build column, the deciding question is whether your team can build the tests and the monitoring that make the system trustworthy, not whether it can call an API. If not yet, partner first and build that capability alongside.

An implementation partner should leave you able to run and change the system without them. That means the code in your repositories, the eval harness and its cases, the runbooks for when it misbehaves, and people on your side who have changed it at least once. In every Bigcircle engagement the work happens in your repositories and you own the code and the IP, so the eval set and its cases stay with you when the engagement ends. Our services page lists what we build, and the comparison of an in-house AI team, an agency and an AI implementation partner covers the choice in more depth.

How do you make a wrong answer easy to catch?

Design for the wrong answer before the right one. A language model produces plausible text, and when plausible and true come apart the result is a hallucination. No model version removes it, so the system around the model either catches it or passes it on. Ground the model in retrieved, cited sources, because a citation a user can click turns a hidden error into one they can check. Where the output feeds another system, use structured output so code can check a typed field; a wrong sentence in a paragraph can only be caught by a careful reader.

Let the system say it cannot answer from the documents it has, and design for that, because a model's default is to answer everything. On the trademark drafting system a citation that does not resolve is the case we designed for. It is rejected and retried against the retrieved evidence, and if it still cannot be supported the claim does not appear in the draft. Then match review to the blast radius: spot checks for internal drafts, sampled review for answers customers see, and approval before sending for anything contractual or regulated.

Build the eval before the prompt, because a prompt tuned without one is tuned to the examples someone happened to try. Start with 50 to 200 real cases from the workflow, each with an expert's verdict on what a good output contains. Grade with code wherever the property is mechanical, and use a model as judge only after checking its verdicts against the expert's on a sample. The guide to LLM evaluation covers the order of those steps. On the DevOps agent, template errors before deploy fell from around 10 percent to 4 percent after we added a validation loop, in our internal measurement. We could say that because the loop recorded every failure.

Language model security differs from ordinary application security because instructions and untrusted data arrive in the same channel, as text. An email, a ticket or a PDF can carry a prompt injection that tries to override your instructions, and no prompt reliably stops it, so enforce permissions outside the model. The DevOps agent has no general shell; it reaches AWS only through a set of purpose-built tools, so no instruction hidden in a request can turn into an arbitrary API call. Put a limit such as "no refunds above 100 dollars" in the API that issues refunds as well as in the prompt, and keep the approval for any irreversible action in code the agent cannot change.

What governance and regulation apply, as of October 2026?

Governance helps a pilot when it makes the approval path known in advance, so a team can ship without waiting for a committee. Set it up as four records before the first pilot starts, because each is easy to keep from day one and hard to reconstruct later:

1. A list of every AI system in use, including features inside vendor products, with its owner and data access.
2. A rule that sets how much review each risk tier gets.
3. A one-page check before each launch: evals passed, adversarial testing done, rollback written, owner named.
4. An incident path naming who is told, what is logged and what shuts the system down.

The regulatory floor, as of October 2026:

Plan against the new dates, and do not read them as a reason to wait. The high-risk dates for Annex III uses such as hiring and credit scoring moved by 16 months from the original 2 August 2026. The duties did not change, and a decision log cannot be written after the fact for decisions a system has already made. Build the decision record into the pilot now, so a system still running in December 2027 already has its history. The same regulation added a ban on generating sexual images of identifiable people without their consent, from 2 December 2026. It also gave systems already on the market until 2 December 2026 to mark AI-generated content under Article 50(2). The guide to building an AI audit trail covers the logging articles and what we put in the record.

How do you set the return before you build?

Write down the metric, its baseline and the date of the verdict before the build starts, along with what result would stop the pilot. Without a baseline, a pilot at the end of its quarter can show activity and cannot show a return. That pilot is the first one cut when a budget tightens. Gartner's January 2026 follow-up counts at least 50 percent of generative AI projects abandoned after proof of concept. Two of the reasons it gives are escalating costs and unclear business value, which is how a pilot with no baseline looks from the finance side. Track cost per completed task next to quality from the first week, because a pilot's inference bill grows with usage and context in a way the demo never showed. The guide to LLM inference covers the levers: a smaller model where evals allow it, model routing, prompt caching and a shorter context window.

We hold our own case studies to the same rule. Every return figure on them is either a formula for your own numbers or a worked example, such as a capacity model that turns 30 recovered hours a month into returned capacity. Apply the same rule in a pilot review: a number counts as a result once someone has measured it.

Lock-in is rarely the model API, which is usually the easiest part of the system to swap. The lasting dependency is everything around it: the eval cases, the retrieval pipeline, the output checks and the traces. Keep those in your own stack. Run your evals against a second provider now and then so switching stays a real option, and be more careful with a provider-specific fine-tune than with plain API calls. Where we do run a fine-tune, on the trademark drafting system, production logging compares the base and fine-tuned models on the same inputs. A regression from a model change then shows up as a score that moved.

What has to change in the team for a pilot to scale?

Settle what stops being done before launch. A process that keeps every old step and adds an AI step costs more than it did, and the AI step is the one people drop first. The conversation most pilots never have is about which steps go, what gets reviewed instead of written, and where the freed hours go. On the trademark drafting system the change was specific: the attorney edits and signs a draft instead of writing from a blank page.

Put the system in the hands of your best domain experts first, because they find failure modes and uses a central team cannot. On the trademark drafting system the attorneys' corrections are the most valuable signal we have, because they measure the gap that automated scoring cannot see. Train people to judge the output rather than to write prompts: what it is reliable for, how to check it, and when to escalate. Show them examples of the system being wrong, not only a highlight reel.

Measure adoption like a product, with weekly use, correction rate and time saved per task, because deployment, adoption and value are three different numbers. And tell people plainly what the tool changes about their job and what it does not.

What does a first quarter look like?

The sequence below fits inside a single quarter, and its purpose is to make the early failures happen on paper. It covers the weeks before a pilot reaches users; moving a pilot that already works in the demo into production is a different job, covered in AI pilot to production.

In weeks 1 and 2, run the readiness check, shortlist three use cases through the three filters, and pick the most forgiving one that still has real value. Write a one-page brief with the workflow today, the workflow after, the metric, the baseline and the stop criteria.

In weeks 3 and 4, domain experts write the eval cases while engineering sets up the test harness and tracing, and security and legal review the data flows. If the eval cases prove impossible to write, you have saved a quarter, because the use case had no ground truth.

Weeks 5 to 10 build the narrowest end-to-end version on real data and real retrieval. Run it in shadow mode against live cases. Grade it weekly against the eval cases and attack it with adversarial inputs before anyone sees it in their workflow.

In weeks 11 to 13, give it to a small group of experts as an assistant and measure quality, adoption, correction rate and cost per task. At the review there are three acceptable outcomes: scale it on the evidence, fix a named gap and test again, or stop it and carry the lesson into the next use case. The one bad outcome is a pilot that keeps running without a verdict.

Which questions should you settle before you sign anything?

If your team can answer each of these in a sentence, the pilot has settled the decisions this guide is about. If three or more draw a blank, that is the work to do first. If the problem is generic and a product you already pay for covers it, most of this list does not apply, and buying is the right call.

1. What workflow does this change, and who owns it today?
2. What is the metric, the baseline, and the number that counts as success?
3. What does a correct output look like, and who can judge one in under five minutes?
4. Do the documents and data exist, and are they accurate and access controlled?
5. What happens when the system is wrong, who catches it, and what does catching it cost?
6. What is in the eval set, and what score allows the launch?
7. What can the system see and do, and what enforces those limits outside the model?
8. Which risk tier is it, which regulations touch it, and who approved it?
9. What does a task cost today, and what happens to that cost if usage grows tenfold?
10. How would you switch model providers, and when did you last test that you can?
11. What changes in the team's process after launch, and who is telling them?
12. What would stop the pilot, and on what date is the verdict?

Common questions

What percent of enterprise AI pilots fail?#

It depends on what the study counted, and no single rate is one to plan against. MIT's 2025 report says 95 percent of organizations get zero return from generative AI, and Gartner says at least 50 percent of generative AI projects were abandoned after proof of concept. They count different things. Both point the same way: a pilot is easy to start and hard to finish when it has no baseline and no way to grade an output.

Is it true that 95% of AI pilots fail?#

Not as stated. MIT NANDA's July 2025 report, labelled preliminary, says 95 percent of organizations get zero return, not 95 percent of pilots. It rests on a review of over 300 public AI initiatives, interviews with people from 52 organizations and a survey of 153 senior leaders, with the results self-reported. Quote it with that method attached, and do not use it to forecast any one pilot.

Why do AI pilots fail to scale when the demo works?#

Because the demo answered an easier question. A demo shows the model can do the task on cases someone picked. A pilot has to show the work gets better or costs less on the cases the workflow actually sends, which needs a baseline, a way to grade an output and a workflow that changes. Most pilots start without those, so they reach their review with activity to show and no verdict.

What are the main enterprise AI adoption challenges?#

We put them down to five decisions made before the code: the problem chosen, how a correct output is judged, whether the grounding data exists, who owns the system once it runs, and whether the workflow around it changes. The model is rarely the blocker. The first decision matters most, because the work's tolerance for error sets what any accuracy figure is worth.

What should we check before choosing use cases?#

Run a readiness check before the roadmap. Find out whether the documents are current and permissioned, and whether an expert can judge an output in a few minutes. Check that the workflow is mapped, what legal and security will allow, whether the team can build tests and monitoring, and whether a named executive will keep the budget through one weak quarter. A low score on any of those changes which use cases are safe to try first.

How many use cases should the first wave include?#

Three at most, with the most forgiving first. Each use case needs its own eval set, integration and review design, so fifty in parallel leaves none of them deep enough to reach production. Filter on value someone in finance would sign, then on whether the data and the ground truth exist, then on what a wrong answer costs. Prefer a workflow your code controls, and save an agent for a task code cannot script.

When should we build the system instead of buying a tool?#

Buy when the problem is generic, such as meeting notes inside a product you already use. Build when the problem is how you compete or depends on your own data, and your team can build the tests and monitoring that tell you whether an output is right. If the team can call an API and cannot yet grade the output, partner first and build that capability alongside.

Sources and further reading

• MIT NANDA, The GenAI Divide: State of AI in Business 2025, July 2025, preliminary findings.
• RAND, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed, August 2024.
• Gartner, Why 50% of GenAI Projects Fail, January 2026.
• Regulation (EU) 2026/1744, the Digital Omnibus on AI, published 24 July 2026.
• AI pilot to production: hardening a pilot that works in the demo and fails at the edges.
• LLM evaluation: the order of steps for the eval set.
• AI audit trail: the record a challenged decision needs.