What we learnedshipping this to production.
Long-form engineering guides on inference, retrieval, agents, evals, and the failure modes teams only meet after launch. Written by the engineers who hit them, with the architecture and the numbers.
41 guides · 590 minutes of reading
In short
Bigcircle's guides each argue for the checks a system needs before anyone trusts it: price an analytics query before it runs, contract-test an MCP server on every change, score a voice call on the audio, and read the real state before an agent says a task is done. Where a point comes from one of our builds, the guide links the case study.
40 guides
- Agentic Systems15 min readGraph engineering: how to design agent graphs that hold up in productionGraph engineering is designing the execution graph an AI system runs in: nodes that do work, edges that decide what runs next, and the shared state that moves along them. Fix the paths that must be predictable, and leave the model free only inside the nodes where judgment helps. Reach for a graph when the work has known stages, approvals, parallel work or different tools per step; a loop is enough for one job.3 Nov 2026Read the guide →
- Agentic Systems16 min readContext graph: what it is, how it differs from a knowledge graph, and how to build one an agent can trustA context graph is a connected record of the facts an AI agent needs before it acts: entities, relationships, where each fact came from, when it was last true, who may see it, and the decisions already taken on it. Unlike a classic knowledge graph, every fact carries provenance, freshness and permissions, and the agent writes back as it works. It does not replace authorization or the system of record.3 Nov 2026Read the guide →
- Agentic Systems15 min readSoftware factory: what a team has to share before agents write the codeA software factory is the shared harness a team runs coding agents through: one registry, one permission model and one set of gates, instead of a copy on every laptop. The older DevOps sense is a pipeline people operate. This guide is the agent sense, and it covers which checks have to sit outside the agent before the team shares them.2 Nov 2026Read the guide →
- Agentic Systems16 min readWhat an MCP gateway does, and when your MCP servers need oneAn MCP gateway is a proxy between your agents and your MCP servers. It registers approved servers, resolves the caller on every request, checks each tool call against policy, holds the upstream credentials, meters calls and spend, and writes the audit line. Add one when a second team, a second client or a server holding credentials arrives, and keep authorization of the record and confirmation of irreversible actions in each system of record.2 Nov 2026Read the guide →
- Agentic Systems16 min readLoop engineering: how to build AI agent loops that stopLoop engineering is designing the checks, caps, triggers and trace reviews that run an AI agent without a person prompting each turn. Enforce the cap and the stop in the harness, check the system the agent changed rather than its transcript, and merge trace-driven changes through an eval gate.1 Nov 2026Read the guide →
- Agentic Systems14 min readSkills vs MCP: what belongs in an Agent Skill, and how to run one in productionSkills vs MCP is the choice between a skill the agent loads into context and an MCP server that holds typed tools under its own credentials. Keep the procedure in the skill, keep governed access on the server, and review the skill file the way you review code.27 Oct 2026Read the guide →
- Reliability & Evaluation13 min readAI SRE agents: what they do on-call, and how to trust one with productionAn AI SRE is an agent that takes the first response to a production alert: it investigates, ranks root-cause hypotheses against telemetry, and proposes a fix or opens a tested pull request. Bigcircle's design puts it on every alert as the first responder, makes each hypothesis carry its evidence, keeps remediation behind scoped tools and a human approval, and sets its autonomy from its record on the client's own replayed incidents rather than a benchmark score.26 Oct 2026Read the guide →
- Strategy & Governance10 min readPII tokenization for LLMs: tokenization vs masking, with codePII tokenization swaps each personal value for a stand-in token before the LLM call and swaps the real value back after the reply. Unlike masking, it keeps the structure the model needs to reason. Bigcircle puts the layer in the request path, adds single-digit milliseconds, and backs it with a CI test that fails if a raw value reaches the provider.22 Oct 2026Read the guide →
- Reliability & Evaluation10 min readGPT-4o deprecation: dates, replacement and how to migrateThe GPT-4o deprecation on OpenAI's API shuts down the listed GPT-4o and GPT-4.1 snapshots on 14 January 2027. OpenAI names GPT-6 Luna as the replacement.19 Oct 2026Read the guide →
- Agentic Systems14 min readAI agent security and sandboxing in productionAI agent security in production means limiting what an agent can reach when it misreads or ignores its permissions, with controls that sit outside the model. Permission checks and guardrails in the prompt still help, but neither stops a network connection the agent opens without asking or a secret it finds in a file. Start with a tool set that is everything the agent can do, and add a sandbox, an egress allowlist, credentials held outside the sandbox and stop rules in code once the agent can run its own code.19 Oct 2026Read the guide →
- Reliability & Evaluation11 min readWhy LLM temperature 0 is not deterministicAn LLM at temperature 0 is not deterministic: the model takes its top-scoring token, but on a shared server that token can change when the batch around the request changes. In one published run, 1000 identical temperature-zero requests came back as 80 different completions. Bigcircle keeps steps with one right answer in code, replays recorded tool results, and checks the state an agent leaves on every repeat.13 Oct 2026Read the guide →
- Agentic Systems14 min readWhen to use multi-agent systems, and how to check the resultMulti-agent systems split one task across several agents with their own contexts, and they pay only when the slices can return without reading each other and the evidence would bury one agent's context. Give each worker a contract the application holds, check each return against a separate read of the expected state before merging, and count every worker on one shared cap. Anthropic measured multi-agent systems at about 15 times the tokens of a chat, so the split has to be worth that.12 Oct 2026Read the guide →
- Agentic Systems12 min readPrompt injection in multi-agent systemsPrompt injection in multi-agent systems spreads when text one agent passes on is read by the next as its task. Keep the forwarded text in a data field, give each agent authority from the person who asked rather than from the agent that called it, and check URLs, paths and references inside the tool that uses them. Hardening each agent on its own does not protect the chain, because published attacks succeed through the hand-offs.8 Oct 2026Read the guide →
- Agentic Systems14 min readMCP server testing before agents depend on itMCP server testing works in two layers. A contract test on every pull request pins the tool list, the arguments, the binding of each state handle to its caller, and the shape of an error, with no model in the run. An agent eval runs on a schedule, and again when the pinned tool list changes, to check that a model still picks the right tool and stops on a refusal.7 Oct 2026Read the guide →
- Agentic Systems14 min readHow much does an AI agent cost to build and run?AI agent cost has three parts, the build, the run and the keep-alive, and the model's token price is one line inside the run. In the worked example here, a ten-turn agent task costs about 24 times one reply at the same token price, because every turn resends the transcript. The guide shows where the build time and the run spend went on Bigcircle's own builds.6 Oct 2026Read the guide →
- Agentic Systems14 min readAgentic workflows vs AI agents, and when not to build oneAgentic workflows put a model in a multi-step process. Use a scripted path when the steps can be written down, and an agent only when the next tool depends on the last result.5 Oct 2026Read the guide →
- Agentic Systems18 min readHow MCP authorization works and what it leaves to youMCP authorization is OAuth 2.1: as of the 2026-07-28 specification, a protected MCP server checks on every request that the token was issued for it, and it must not pass the user's token through to another service. The specification stops there, so which tool may run, with which arguments, for which person, is still your code. The MCP security best practices that follow are to keep the application secret out of the model, mint a token per run, check each call before it leaves, return a refusal the model can repeat, and record both outcomes.4 Oct 2026Read the guide →
- Agentic Systems14 min readAI agent memory isolation for multi-user agentsAI agent memory in a multi-user agent has to be keyed to the acting user inside the tenant, or one person's notes and OAuth grants show up in a Slack group or another customer's workspace. A scoped tool token does not prevent that, because a scope limits operations and says nothing about whose request it is. Key memory and credentials to the user and the tenant, and decide in code what a direct message may carry that a channel may not.3 Oct 2026Read the guide →
- Reliability & Evaluation13 min readVoice agent testing beyond the transcriptVoice agent testing has to score the audio as well as the transcript, because a correct transcript can still be a failed call. Run voice AI evals for turn-taking, interruptions, intelligibility and silence on the recording, keep those verdicts apart from the text score, and prove a fix by replaying the same recorded calls.2 Oct 2026Read the guide →
- Reliability & Evaluation12 min readAI agent monitoring in productionAI agent monitoring watches a production agent on two layers that answer different questions: a quality layer that scores a sample of sessions, and an infrastructure layer that watches every model call, tool call and permission check. Agent observability is the trace both layers read, one span per step under a shared trace ID, and it goes further than LLM observability by recording the tool calls and permission decisions around each model call. Checks whose failure must never reach a user run inline, and every failure that escapes becomes a test case or a scheduled probe.26 Sep 2026Read the guide →
- AI Applications15 min readDocument intelligence for AI agentsDocument intelligence for an AI agent means checking extracted fields across the whole packet before anything is written: the same party, the same period, figures that agree and no missing document. Intelligent document processing reads each document and returns its fields, and a lending, claims or compliance decision depends on how those fields relate. Bigcircle resolves every party to one entity, reconciles the figures, keeps a page and span on every value, and passes a policy gate before any write. Field accuracy per document measures none of those steps, so the eval scores the case.26 Sep 2026Read the guide →
- Agentic Systems14 min readHow to build a voice AI agentTo build a voice AI agent, choose a cascade, a speech-to-speech model or a managed pipeline by how much of each turn you need to see and time, not by how fluent the demo sounds. On our insurance voice agent, response fell from 2.5 to 3 seconds to around 900 milliseconds, and most of the work went into deciding when the caller had finished. Whichever architecture runs the audio, the tools that move money or change a record stay in your code, with their checks and the handoff to a person.26 Sep 2026Read the guide →
- Agentic Systems17 min readText to SQL for analytics agents in productionText to SQL turns a business question into a database query, and an analytics agent built on it fails quietly, because the warehouse accepts a wrong query and the model writes a fluent paragraph around the wrong number. Bigcircle puts governed metric definitions before the SQL, parses and prices every query before it runs, keeps the hard limits in the database, and blocks any number in an answer that cannot be matched to a cell in the result. Public text-to-SQL scores, such as the 82.95% that led BIRD in September 2026, measure none of those steps, so the eval set comes from questions the business already asks.24 Sep 2026Read the guide →
- Agentic Systems17 min readHow to introduce TypeSafe's Jev into agentic workflowsJev is TypeSafe AI's decision model: it answers typed questions about a state with a probability for each option, and it writes no text and runs no tool. Bigcircle recommends introducing it into an agentic workflow as a shadow gate that scores tool calls and blocks nothing, then letting its answers drive low-risk calls, model choice, tool shortlists and loop checks, one step at a time. Jev is never the only decider for an action that cannot be undone.24 Sep 2026Read the guide →
- Agentic Systems13 min readHow to build a semantic layer for AI agentsA semantic layer for AI agents defines each business term once, with an owner and a version, and serves it to every agent through one governed MCP entry point. Some teams call it a context layer for AI agents, because it covers documents, synonyms and access rules as well as metrics. The entry point runs each call as the user, cites its sources and logs every call, so separate agents stop giving conflicting numbers for the same question.24 Sep 2026Read the guide →
- Agentic Systems14 min readHow to design MCP tools for agentsMCP tools are the operations an MCP server exposes for a model to call, and the model chooses among them from their definitions. When several sources answer the same questions, put one consolidation MCP between your agents and the vendors. Expose fewer, larger tools that return structured JSON, authenticate every request as the user, rate-limit by caller, and read the call log to decide the next batch mode or schema change.24 Sep 2026Read the guide →
- Agentic Systems15 min readMCP prompts vs tools for agent playbooksMCP prompts are templates a server offers for a user to pick, such as a slash command, and the client fetches one with prompts/get. A workflow an agent should find and run on its own belongs in the other primitive: write it as a playbook with a typed contract, expose it through MCP as a tool, and put three meta-tools in front of the catalog: search, get the schema, execute. Keep central and repo-local playbooks in one catalog, and log every run.24 Sep 2026Read the guide →
- Strategy & Governance13 min readHow to move an AI pilot to productionTo move an AI pilot to production, run it on real inputs and sort the failures first, because a pilot that fails only at the edges can be hardened and one that fails at the core task needs a rebuild. Then build the eval set before changing anything, put checks in code before an agent acts, measure cost and latency against that eval set, and write acceptance as a test someone can run after handover. Bigcircle sells this work as production hardening, and the client owns the code in every engagement.22 Sep 2026Read the guide →
- Strategy & Governance12 min readAI implementation partner vs agency vs in-house teamBring in an AI implementation partner when your first production AI system has to run before you can hire for it, and keep its code and eval set in your repositories so your own team can take it on. Build in-house when the system is part of what you sell and someone on your staff can already review AI work. Use an AI development agency when the work can be finished and scored against a test you wrote.22 Sep 2026Read the guide →
- Reliability & Evaluation18 min readHow to build an AI audit trailAn AI audit trail links each decision an agent makes to the version of the evidence it used, the business rule that applied, what the reviewer changed, and a separate read showing the target system stored the result. A tool trace shows what ran, and stops explaining the decision once the linked document changes. The EU AI Act logging requirements in Articles 12, 19 and 26 now apply from 2 December 2027 for Annex III systems, and they ask for event logs rather than this record.9 Sep 2026Read the guide →
- Reliability & Evaluation27 min readHow to build an LLM evaluation from real failuresLLM evaluation, or LLM evals, means testing a model's answers with checks that take a judgement, such as grounded, correctly cited or correctly refused. You cannot write the rubric first: criteria emerge from reading your own failures, and everything downstream depends on doing that step properly.1 Sep 2026Read the guide →
- AI Applications15 min readRAG vs fine-tuning vs agentic search, and when to use eachRAG vs fine-tuning comes down to the failure you can see: use retrieval when the model is missing a fact, and fine-tuning when the facts are right and the form is wrong. Agentic search, also called agentic RAG, is for questions no single query can answer, and every extra step is another model call the user waits for. Exhaust prompting before any of them.1 Sep 2026Read the guide →
- Strategy & Governance14 min readAI agent governance and autonomy levels in practiceAI agent governance, in practice, is deciding which actions an agent may take without a person, on what evidence, and who approves each change. Set the autonomy level per action rather than per agent, because a read and a refund do not carry the same consequence. Move an action up only when a check in code catches what the approver was catching, and write down at the same time what moves it back. On Solarpunk, plans completed without a person rose from about 55% to 80% while the agent's authority stayed the same.13 Aug 2026Read the guide →
- Reliability & Evaluation13 min readAI agent verification and testing before work counts as doneAI agent verification means reading the result of an action from the system it changed before the task counts as done, because an agent that treats a tool's reply or its own account as proof will report unfinished work as done. In a 2026 study of 20,574 coding-agent sessions, inaccurate self-reporting appeared in 22.58% of the validated misalignment episodes. Decide before the action what finished looks like, read it back from the system the action changed, and treat what you cannot read as unknown.13 Aug 2026Read the guide →
- Agentic Systems13 min readHow to set AI agent permissions and access controlSet AI agent permissions one action at a time, for one workflow, rather than approving the agent as a whole. Give each change its own tool, and run the authorization check in code before the call, against the person who asked, the target and its current state. Let the agent keep its own identity while it acts for that person, and attach the credential outside the model.13 Aug 2026Read the guide →
- Agentic Systems12 min readHuman in the loop AI agents and when a person must approveHuman in the loop AI agents put a person in front of a write when the change is hard to undo, when nothing but the agent can confirm the result, or when there is no record yet of how the agent handles that action. Show the approver the exact change, enforce the approval in the tool that makes it, and remove approval one action at a time once the record and an independent check support it.13 Aug 2026Read the guide →
- Strategy & Governance17 min readWhy AI pilots fail in the enterpriseAI pilots fail mostly on decisions made before the code: the wrong problem, no way to judge a correct output, missing data, no owner, and a workflow that never changes. The MIT finding that 95 percent of organizations get zero return from generative AI rests on interviews with 52 organizations and a survey of 153 senior leaders, so it describes the field rather than your pilot. This guide argues why each failure happens and what Bigcircle built on its own systems to stop it deciding the outcome, then turns that into checks to run before budget goes in.12 Jun 2026Read the guide →
- Agentic Systems12 min readHow to build an AI agent for productionTo build an AI agent for production, decide whether the task needs one, pick the model on your own eval cases, give it typed tools and no shell, keep credentials out of its context, check every step that changes state, bound the retries, and keep a trace a person can read. On one of our desktop agents, grounding in the user's workspace took plans finished without a person from about 55% to 80%.12 Jun 2026Read the guide →
- AI Applications22 min readRAG in production: why it fails and how to find the broken layerRAG in production usually fails quietly: the answer sounds right, cites a nearby passage, and misses the fact the user needed. Retrieval, context assembly and generation break for different reasons, so measure them separately before changing a model or a prompt. A long document is only as answerable as the chunks it was split into.12 Jun 2026Read the guide →
- AI Infrastructure10 min readHow LLM inference works and how to optimize itLLM inference is running a trained model on a prompt to produce an answer, in two phases: prefill reads the prompt and sets the wait for the first token, and decode writes the answer one token at a time and sets how fast it streams. On our insurance voice agent, end-to-end response fell from 2.5 to 3 seconds to around 900 milliseconds, and the model call was one stage of that budget among several.12 Jun 2026Read the guide →
Common questions
What are Bigcircle's guides for?
Bigcircle's guides are long-form engineering guides to inference, retrieval, agents, evals, and the failures teams meet after launch. Each one argues for a specific check and the order to run it in, and links the case study where the point comes from our own work. You pick the failure you are debugging, and the guide names the mechanism, the check, and where that check is easy to skip.
What should be checked before trusting an analytics agent?
Text to SQL for analytics agents in production sets out the checks Bigcircle recommends. Parse every query before it reaches the warehouse, price the plan and block a scan that is too large before the real query runs, and execute read-only as the asking user under a timeout and byte caps. Every number in the answer cites its query, and the reply abstains when the rows do not support the claim.
How should you test an MCP server?
Bigcircle recommends a contract test, set out in MCP server testing before agents depend on it, on every pull request with no model in the loop. The test pins the tool list, the arguments, the authorization behaviour, and the error shape, so the pull request fails when a reply drifts. A scheduled agent eval asks whether a model picks the tool, recovers from a typed error, and stops when the tool refuses.
When should I read the Strategy & Governance guides?
Read Bigcircle's Strategy & Governance guides when the decision is whether a pilot should reach production and how far an agent may act on its own: Why AI pilots fail in the enterprise and AI agent governance and autonomy levels in practice. For the call on whether to build an agent at all, read Agentic workflows vs AI agents, and when not to build one and How much does an AI agent cost to build and run?.
Every one of these is a problem we hit in production first.
The concepts these guides assume.
Agentic Systems
AI AgentTool Calling (Function Calling)ReAct Agents (Reason + Act)CodeActPlan-and-ExecuteComputer Use Agents (GUI Agents)Agent MemoryMulti-Agent OrchestrationModel Context Protocol (MCP)LLM Observability & TracingLLM GuardrailsPrompt InjectionContext WindowStructured Output & Constrained DecodingEvals & LLM Evaluation MetricsAI Software FactoryRetrieval-Augmented Generation (RAG)AI Hallucination & GroundingGroundedness (Faithfulness)Decision ModelConstrained DecodingRealtime Voice AgentsSpeech-to-Text (ASR)Text-to-Speech (TTS)Time to First Token vs Time Between TokensReasoning Models & Extended ThinkingPrompt CachingLoop EngineeringReflexion & Self-CorrectionPrompt Engineering & System PromptsMeta HarnessGraphRAG
AI Applications
Retrieval-Augmented Generation (RAG)EmbeddingsChunking StrategiesVector Search & Vector DatabasesHybrid SearchRerankingLate Interaction & ColBERTGraphRAGContext WindowPrompt InjectionEvals & LLM Evaluation MetricsAI Hallucination & GroundingSupervised Fine-Tuning (SFT)LoRA & QLoRA Fine-TuningFew-Shot Prompting & In-Context LearningStructured Output & Constrained DecodingConstrained DecodingPlan-and-ExecuteReAct Agents (Reason + Act)LLM-as-a-JudgePrompt CachingModel DistillationSynthetic DataModel Routing & CascadingKnowledge CutoffGroundedness (Faithfulness)Vision-Language Models (VLMs)Tool Calling (Function Calling)LLM GuardrailsAI Agent
Reliability & Evaluation
AI AgentTool Calling (Function Calling)Evals & LLM Evaluation MetricsLLM Observability & TracingLLM GuardrailsComputer Use Agents (GUI Agents)Structured Output & Constrained DecodingLLM-as-a-JudgeAI Hallucination & GroundingRetrieval-Augmented Generation (RAG)RerankingHybrid SearchBenchmark ContaminationSynthetic DataPrompt InjectionSupervised Fine-Tuning (SFT)LLM Temperature & SamplingChunking StrategiesGroundedness (Faithfulness)Judge Alignment (TPR, TNR, Cohen's Kappa)Realtime Voice AgentsSpeech-to-Text (ASR)Text-to-Speech (TTS)Time to First Token vs Time Between TokensLogprobs & ConfidenceContinuous BatchingModel Routing & CascadingMulti-Agent Orchestration
Strategy & Governance
Evals & LLM Evaluation MetricsAI Hallucination & GroundingLLM GuardrailsLLM Observability & TracingRetrieval-Augmented Generation (RAG)AI AgentPrompt InjectionModel Routing & CascadingLLM-as-a-JudgeLLM Red Teaming & JailbreaksTool Calling (Function Calling)Prompt CachingData PoisoningStructured Output & Constrained DecodingComputer Use Agents (GUI Agents)Supervised Fine-Tuning (SFT)Context Window
Let's build something that ships.
Tell us what you're building. We'll tell you whether it needs a fixed-scope build, a prototype taken to production, or a programme we run, what's achievable, and where the real bottlenecks are.