What we learnedshipping this to production.
Long-form engineering guides on inference, retrieval, agents, evals, and the failure modes teams only meet after launch. Written by the engineers who hit them, with the architecture and the numbers.
28 guides · 476 minutes of reading
At a glance
The checks Bigcircle runs before a system is trusted: parse and price every analytics query, contract-test an MCP server on every change, score a voice call on the audio, and read the expected state before an agent may say it is done.
Pick the failure you are debugging. The guide names the mechanism and the check, then the places that check is easy to skip.
- Who this is for
- Engineers and technical leads working on AI system design, quality, and operation.
- Topics
- Retrieval
- AI agents
- Evaluation
- Inference
In short
Bigcircle's guides are the checks run before a system is trusted: parse and price every analytics query, contract-test an MCP server on every change, score a voice call on the audio, and read the expected state before an agent may say it is done. They are written from problems we hit in production first. You pick the failure you are debugging, and the guide names the mechanism, the check, and the places that check is easy to skip.
27 guides
- Agentic Systems16 min readTesting MCP servers before agents depend on themTest an MCP server in two layers before an agent depends on it. A contract test on every pull request pins the catalogue, the arguments, the session, and the error shape. An agent-in-the-loop eval runs on a schedule, or when that catalogue changes.7 Oct 2026Read the guide →
- Agentic Systems14 min readWhat a production AI agent costs to build and runA production agent costs the build, the run, and the keep-alive. The model line moves after launch, and a cap that only sends an email has already paid for the calls in between.6 Oct 2026Read the guide →
- Agentic Systems14 min readWhen not to build an AI agentA workflow is enough when the steps are already written down. Retrieval is enough when one query can find the passage. An agent earns the loop only when the next tool depends on a result you could not name in advance.5 Oct 2026Read the guide →
- Agentic Systems16 min readMCP tool access in production: scoped tokens, audit, and refusalAn MCP connection is not a place to put the application secret. Mint a short-lived token for the run, check policy before the call leaves, and keep an audit line for every allow and every refusal.4 Oct 2026Read the guide →
- Agentic Systems16 min readMulti-user agent memory and credential isolationA scoped tool token does not stop one user's memory or OAuth from reaching a Slack group or another tenant. Key memory and credentials to the acting user inside the tenant, and decide in code what a direct message may carry that a channel may not.3 Oct 2026Read the guide →
- Reliability & Evaluation15 min readVoice AI evals: conversation audio, not just transcriptA correct transcript can still be a failed call. Score turn-taking, interruptions, intelligibility and silence on the audio, keep those verdicts off the text score, and prove a fix by replaying the same recorded calls.2 Oct 2026Read the guide →
- Reliability & Evaluation13 min readDual-layer production agent ops: quality evals vs infrastructure investigationRun a production agent on two layers that answer different questions. The quality layer asks whether the user got what they needed, using evals on a sample of traces. The infrastructure layer asks whether every model call, tool call and permission check ran as intended, and investigates when one did not. Anything that must never reach a user runs inline on every turn, both layers read one trace, and every escaped failure ends as an eval case or a probe, with personal data removed first.26 Sep 2026Read the guide →
- AI Applications14 min readDocument intelligence beyond extraction: cross-document reasoning for agentsExtraction reads each document; a decision needs the whole packet. Before an agent acts on documents, resolve every party to one entity, reconcile the figures that should agree, check the packet is complete, keep a page and span on every value, validate in code, and pass a policy gate that decides between writing, review and asking for a missing document. Field accuracy per document says nothing about any of those steps, so evaluate the case, not the field.26 Sep 2026Read the guide →
- Agentic Systems15 min readProduction voice-agent architecture: cascade vs streaming vs managedChoose a voice-agent architecture on your latency budget and on how much control you need at each stage, not on how fluent the demo sounds. A cascade gives you text and a check at every step, a speech-to-speech model gives the most natural turn-taking with the least visibility, and a managed pipeline hands the audio path to a vendor while you keep the tools. In all three, code owns turn-taking rules, read-back confirmation before irreversible actions, and a warm handoff with a context packet, and we design the handoff first.26 Sep 2026Read the guide →
- Agentic Systems31 min readBuilding production-grade analytics agentsA production analytics agent is a layered system: intake, a governed metric layer, planning, SQL generation, validation gates, execution, result shaping and answer synthesis, with memory and a tool boundary around them. Its minimum core parses every query, prices it with EXPLAIN, runs it read-only under caps, and filters the answer so every number cites its query. A bounded workflow orchestrates it, every step is traced and replayable, changes pass eval gates, and people review only the cases that earn it. Jev judges, code enforces, and the model generates.24 Sep 2026Read the guide →
- Agentic Systems15 min readIntroducing Jev into agentic workflowsJev is TypeSafe's System One model: state in, Choice, Score or Noul answers out, each with calibrated probabilities. We introduce it as a shadow tool gate first, then a soft gate, a model router, catalogue selection, and loop and completion supervisors. It is never the only decider for an irreversible action.24 Sep 2026Read the guide →
- Agentic Systems14 min readShared business context for every AI agentAgent sprawl without shared definitions produces conflicting numbers. Define business terms once, each with an owner and a version, and serve retrieval through one governed MCP entry point that runs as the user, cites its sources, and logs every call.24 Sep 2026Read the guide →
- Agentic Systems15 min readMCP tool design: fewer, larger toolsWhen several sources answer the same questions, put one consolidation MCP between your agents and the vendors. Expose fewer, larger tools that return structured JSON, authenticate every request as the user, rate-limit by caller, and read the call log to decide the next batch mode or schema change.24 Sep 2026Read the guide →
- Agentic Systems16 min readAgent playbooks as MCP toolsWrite each recurring workflow as a playbook with a typed contract, expose it through MCP as a tool, and put three meta-tools in front of the catalog: search, get the schema, execute. Keep central and repo-local playbooks in one catalog, and log every run.24 Sep 2026Read the guide →
- Strategy & Governance19 min readEmbed vs CTO-led pod vs fixed-scopeEmbed a Bigcircle engineer when you can direct the work and the gap is AI depth in your repository. Take a CTO-led pod when you have a roadmap and no senior AI leadership to run it. Buy a fixed-scope build when you want the working system, and not the hiring.22 Sep 2026Read the guide →
- Strategy & Governance19 min readEmbedded AI engineer vs hire vs agencyEmbed a Bigcircle engineer when you need AI depth in your repository without a permanent headcount bet. Hire full-time when AI is a multi-year competency you can manage. Use an agency for a bounded deliverable with acceptance tests, and not for ongoing ownership inside your systems.22 Sep 2026Read the guide →
- Reliability & Evaluation22 min readHow to build an audit trail for AI decisionsA tool trace can show what ran without explaining the accepted business decision. This guide defines the record an AI workflow needs when a case is challenged or its evidence changes.9 Sep 2026Read the guide →
- Reliability & Evaluation27 min readHow to build LLM evalsYou cannot write the rubric first. Criteria emerge from reading your own failures, and everything downstream depends on doing that step properly.1 Sep 2026Read the guide →
- AI Applications15 min readRAG vs fine-tuning vs agentic searchExhaust prompting first. Use retrieval when the model is missing a fact, fine-tuning when the facts are right and the format is wrong, and agentic search only when no single query can find the answer.1 Sep 2026Read the guide →
- Strategy & Governance11 min readHow to roll out AI agent autonomy in five levelsPromote one action class by one authority level, and only after that action has the exit evidence the change requires. A drift or an incident demotes the action. A read, a payment, and an infrastructure change do not share a level.13 Aug 2026Read the guide →
- Reliability & Evaluation11 min readWhy AI agents report unfinished work as doneA successful API response is not a finished task. A run may report done only after a separate read matches the expected state, including the downstream effects the contract named.13 Aug 2026Read the guide →
- Agentic Systems11 min readHow to scope AI agent permissionsApproving a whole agent accepts consequences that are not defined yet. Bind one action to a principal, task, target, policy, credential, approval, expected state, and recovery path. Bigcircle calls that contract an Agent Action Boundary.13 Aug 2026Read the guide →
- Agentic Systems10 min readWhen your AI product needs write accessWrite access is the point where a wrong answer becomes a wrong change to money, access, or production state. The next release should ask for the smallest authority that can finish the task, and that boundary should be reviewable.13 Aug 2026Read the guide →
- Strategy & Governance26 min readWhy enterprise AI pilots failThe choices that sink a pilot are made before the code: which problem to pick, what data grounds it, how you measure it, who owns it when it misbehaves, and what happens to the workflow around it. The published failure studies disagree on the yardstick, so the table comes before the headline.12 Jun 2026Read the guide →
- Agentic Systems18 min readHow to build production AI agentsAn agent is a loop that observes, acts through tools, and stops. The production question is which decisions the model may make, which code must make, and where a person approves the risk.12 Jun 2026Read the guide →
- AI Applications31 min readWhy RAG systems fail in productionMost RAG failures look like model failures until retrieval, context assembly, and generation are measured separately. A plausible answer with a nearby citation can still be the wrong passage.12 Jun 2026Read the guide →
- AI Infrastructure18 min readHow LLM inference worksInference splits into compute-bound prefill and memory-bound decode. Cache, batching, and kernel choice follow from that split, and they are tuned for the product's latency rather than for a short benchmark.12 Jun 2026Read the guide →
Common questions
What are Bigcircle's guides for?
They are long-form engineering guides to inference, retrieval, agents, evals, and the failure modes teams meet after launch. Bigcircle writes them from problems hit in production first, with the architecture and the numbers. You pick the failure you are debugging. The guide names the mechanism, the check, and the places that check is easy to skip.
What does Bigcircle check before trusting an analytics agent?
Bigcircle parses every query into an AST before it reaches the warehouse. An EXPLAIN cost gate prices the plan and blocks a scan that is too large, before the real query runs. Execution is read-only, under a timeout and byte caps, as the asking user. The answer filter requires every number to cite its query, and the reply abstains when the rows do not support the claim.
How does Bigcircle test an MCP server?
Bigcircle runs a contract test on every pull request, with no model in the loop. The test pins the catalogue, the arguments, the session, and the error shape, and it fails the pull request when the reply drifts. An agent eval runs on a schedule, or when the tool list changes, and asks whether a model picks the tool, recovers from a typed error, and stops when the tool refuses.
When should I read the Strategy & Governance guides?
Read them when the decision is who does the work and which engagement you buy. Bigcircle files Embedded AI engineer vs hire vs agency and Embed vs CTO-led pod vs fixed-scope under Strategy & Governance, with Why enterprise AI pilots fail and How to roll out AI agent autonomy in five levels.
Every one of these is a problem we hit in production first.
The concepts these guides assume.
AI Applications
Retrieval-Augmented Generation (RAG)EmbeddingsChunkingVector Search & Vector DatabasesHybrid SearchRerankingLate Interaction & ColBERTGraphRAGContext WindowPrompt InjectionEvalsHallucination & GroundingFine-Tuning (SFT)LoRA & QLoRAFew-Shot & In-Context LearningStructured Output & Constrained DecodingConstrained DecodingPlan-and-ExecuteReAct (Reason + Act)LLM-as-a-JudgePrompt CachingDistillationSynthetic Training DataModel Routing & CascadingKnowledge CutoffGroundedness (Faithfulness)Vision-Language Models (VLMs)Tool / Function CallingGuardrailsAI Agent
Agentic Systems
AI AgentTool / Function CallingReAct (Reason + Act)CodeActPlan-and-ExecuteComputer Use & GUI AgentsAgent MemoryMulti-Agent OrchestrationModel Context Protocol (MCP)LLM Observability & TracingGuardrailsPrompt InjectionEvalsStructured Output & Constrained DecodingContext WindowRetrieval-Augmented Generation (RAG)Hallucination & GroundingGroundedness (Faithfulness)Constrained DecodingRealtime Voice AgentsSpeech-to-Text (ASR)Text-to-Speech (TTS)TTFT vs TBT (Latency Metrics)Reasoning & Extended ThinkingPrompt Caching
Reliability & Evaluation
AI AgentTool / Function CallingEvalsLLM Observability & TracingGuardrailsComputer Use & GUI AgentsStructured Output & Constrained DecodingLLM-as-a-JudgeHallucination & GroundingRetrieval-Augmented Generation (RAG)RerankingHybrid SearchBenchmark ContaminationSynthetic Training DataPrompt InjectionFine-Tuning (SFT)Temperature & SamplingChunkingGroundedness (Faithfulness)Judge Alignment (TPR, TNR, Kappa)Realtime Voice AgentsSpeech-to-Text (ASR)Text-to-Speech (TTS)TTFT vs TBT (Latency Metrics)
Strategy & Governance
EvalsHallucination & GroundingGuardrailsLLM Observability & TracingRetrieval-Augmented Generation (RAG)AI AgentPrompt InjectionModel Routing & CascadingLLM-as-a-JudgeRed-Teaming & JailbreaksFine-Tuning (SFT)Tool / Function CallingPrompt CachingData PoisoningStructured Output & Constrained DecodingModel Context Protocol (MCP)
Let's build something that ships.
Tell us what you're building. We'll tell you whether you need an engineer embedded or the whole build led, what's achievable, and where the real bottlenecks are.