How should a production voice agent be built? We choose between three architectures on the latency budget and on how much control each stage needs. A cascade runs speech to text, a language model and text to speech as separate stages. A speech-to-speech model hears and answers in one step. A managed pipeline hands the whole audio path to a vendor. Whichever we pick, code owns turn-taking, read-back confirmation before any irreversible action, and a warm handoff to a person, and the handoff is designed first.
The hard part is that a voice agent fails in real time, in front of the caller, with no chance to edit. A chat agent that pauses for two seconds looks like it is thinking. A voice agent that pauses for two seconds sounds broken, and the caller starts talking again just as the reply begins, so both sides talk over each other and the turn is lost.
The second difficulty is that the demo tests the wrong things. A demo call is quiet, scripted and short, with a caller who waits politely for each answer. Real callers talk over the agent, switch language mid-sentence, call from cars and kitchens, and ask to change an order they placed an hour ago. Fluency in the demo tells you little about any of that, and it tells you nothing about what happens when the agent must stop and hand over.
Speech also removes the review step people rely on elsewhere. On a screen, a user can read a proposed refund before pressing a button. On a call, the caller hears it once, and a model that says "done" before the tool has returned has made a claim nobody can check.
This guide sets out the three architectures first, then how we choose between them. It then covers where the latency budget goes, turn-taking and barge-in, the handoff, voice confirmation for irreversible actions, and where a decision model such as Jev can help. It closes with logging, testing and release.
What are the three ways to build a voice agent?
A voice agent turns caller audio into a reply and, often, into an action in another system. The three architectures differ in how many stages that path has and who runs each one. The difference decides what you can inspect, what you can tune, and what you can log when a call goes wrong.
This figure is a framework. It compares architecture shapes and says nothing about any vendor's latency, accuracy or price.
A cascade chains three models: streaming speech to text, a language model working on text, and streaming text to speech, with a turn detector deciding when the caller has finished. Every boundary between stages produces text you can read, log and check. You can swap one stage without touching the others, and a regulated disclosure can be checked as a string before it is spoken.
A speech-to-speech model takes audio in and produces audio out from one model, calling tools by function call along the way. It handles interruptions and tone more naturally, because nothing has to be converted to text and back between turns. The cost is visibility. The text transcript is a side output rather than the thing the model reasoned over, so a check on the transcript is a check on a copy.
A managed pipeline is usually a cascade that someone else runs. The vendor operates transcription, turn detection, the model call, voice, telephony and session state behind one connection and one bill. Your code supplies the prompt, the tools and the policy around them, and sees only what the vendor chooses to expose about each stage.
| Cascade | Speech to speech | Managed pipeline | |
|---|---|---|---|
| Stages you run | All of them | The model call and tools | Tools and policy |
| Text at each step | Yes, by construction | Side transcript only | What the vendor exposes |
| Swap one component | Yes | No | Only inside the vendor's options |
| Turn-taking | Yours to build and tune | Mostly inside the model | The vendor's, with settings |
| Best fit | Regulated actions, strict wording, many languages | Open conversation where tone matters most | A first launch, or a team without voice engineers |
| Typical failure | Latency added at each hand-off between stages | An action taken on audio nobody can inspect | A limit you cannot see past when a call fails |
How do you choose between them?
We choose on three questions, asked in this order. What latency budget does the use case allow, what must be checked or logged at each stage, and who will own turn-taking when it goes wrong? Demo fluency is not on the list, because every architecture can sound good on a quiet, scripted call.
The latency budget comes from the caller's patience, measured as the gap between the end of their speech and the first sound of the reply. A support line answering "where is my order" can hide a short pause behind a spoken acknowledgement. A line where callers interrupt constantly, or speak two languages in one sentence, needs a tighter loop, and the architecture has to leave room for tool calls inside it.
The control question is usually decided by the actions the agent can take. If the agent reads a policy renewal date, a speech-to-speech model with a read-only tool is often fine. If it moves money, cancels an order or changes a record under regulation, we want text at each step. That lets code check the wording of a disclosure and the arguments of the tool call before either happens.
Ownership is the question teams skip. A managed pipeline moves turn detection, noise handling and telephony to a vendor, which saves months of work and also moves the fix for a dropped turn onto someone else's roadmap. AssemblyAI's own guidance on its Voice Agent API puts the trade in these terms: build your own stack when you need control of each component, and use a single API when launch time and a smaller integration count matter more.
The vendors now offer both paths from one product, which makes the choice reversible. Microsoft Foundry's Voice Live runs prompt agents as a managed voice path and, in its July and August 2026 update, lists hosted agents with Voice Live as generally available for teams that bring their own runtime. AssemblyAI sells the full pipeline and its streaming transcription as separate lanes. Our usual advice is to start where the tool boundary is easiest to own and move a stage in-house only when logs show it failing.
| If the use case has | We usually start with | Because |
|---|---|---|
| Irreversible actions, required disclosures, audit | Cascade | Code can check text before it is spoken and arguments before a tool runs |
| Open conversation, coaching, low-stakes reads | Speech to speech | Turn-taking and tone matter more than inspecting each step |
| A first launch and no voice team | Managed pipeline | Telephony and turn detection are solved problems you do not have to rebuild |
| Callers mixing languages in one sentence | Cascade, or managed with a proven transcription model | Turn detection and transcription fail first on code-switched audio |
Where does the latency budget go?
The budget is spent across every stage of a turn, and the stage that decides the most is the one teams tune last. The naive assumption is that the language model is the slow part. In our own voice work the component that broke first, and set the floor for everything else, was the decision about when the caller had finished speaking.
This figure is a framework showing the order of stages in one cascade turn. It carries no timings, because those depend on your models, network and telephony.
On AVOX, an insurance voice agent we built and tested internally but which was not deployed, end-to-end response went from 2.5 to 3 seconds down to around 900 milliseconds on our own audio path. Turn detection took the most work. A saving of a few hundred milliseconds in speech synthesis is invisible next to a detector that waits half a second too long to decide the caller has stopped.
Three changes did most of the work, and none of them was a faster model. We loaded the policy and claim context when the call opened rather than fetching it mid-turn, which took the largest variable cost off the critical path. The stages streamed into each other instead of waiting for a complete result. We also tuned the turn detector for code-switched, noisy calls rather than for quiet test recordings.
Tool calls are the other place budgets go. A lookup that takes a second on a web page is a second of silence on a call. We keep read tools fast and narrow, and we let the agent speak a short acknowledgement while a slower tool runs. That acknowledgement must never claim the result, and a spoken "let me check that" is honest where "your refund is done" before the tool returns is not.
Measure the budget per turn, as the gap between the end of the caller's speech and the first audio of the reply, and keep the distribution rather than the average. A voice agent with a good mean and a long tail still has callers who hear a three-second silence, and those are the calls that end in a complaint. The latency metrics page covers the terms we use for each part.
How should turn-taking and barge-in work?
Turn-taking decides when the caller has finished and the agent may speak. Barge-in decides what happens when the caller speaks while the agent is still talking. Both are state problems more than audio problems, and treating them as audio problems is why many voice agents feel rude in production.
A silence threshold is the simplest end-of-turn detector: wait for a gap of a set length and assume the caller is done. It works on quiet, fluent speech. On a call where the caller pauses to find a policy number, or switches between English and Hindi mid-clause, the natural pauses fall in the wrong places. The agent either cuts in or waits through a pause that was not an ending.
We combine voice activity detection, which says whether anyone is speaking, with a model that judges whether the words so far form a finished turn. The managed pipelines offer the same combination under their own names. Whichever you use, test it on your own callers' audio, because a detector tuned on read speech tells you little about a caller reading a claim number from a letter.
Barge-in needs three steps, and missing any one of them breaks the conversation. Playback stops at once, so the agent does not keep talking over the caller. The conversation history is trimmed to what the caller actually heard, because the model otherwise believes it said a full sentence the caller cut off. The interruption is then classified as a backchannel such as "right" or "mm", or as a new turn that changes the request.
History trimming is the step most often missed. If the agent was reading three delivery options and the caller interrupted after the first, the model's history still holds all three. On the next turn it may refer to an option the caller never heard. Microsoft lists this as auto-truncation in its Voice Live release notes, and a cascade has to build the same thing by hand.
Barge-in stops speech. It does not stop an action. If a tool call was already sent when the caller interrupted, it is running, and the interruption changes nothing about its effect. That is the reason confirmation has to come before an irreversible call, which the section after the handoff covers.
Why design the handoff first?
A production voice agent will reach calls it should not handle, and the quality of the handoff decides what the caller thinks of the whole system. We design it before the happy path. The alternative is a handoff that exists only as a last-minute transfer to a queue, with the caller repeating everything to a person who knows nothing.
Yuma's announcement of its voice agent in September 2026 makes the same argument from the vendor side. Its founder, Guillaume Luccisano, is quoted saying the company built the handoff paths before anything else, and the product offers a direct transfer, a callback with a ticket, and an explicit referral to another channel. Voiso describes its AI voice agents as transferring warm, with the conversation context travelling with the call to a queue, an agent or a number.
This figure is a framework. The triggers and packet fields are the ones we design for, not a vendor's feature list, and the figure reports no measured result.
The triggers belong in code. A caller asking for a person is the obvious one, and it should always work, on the first request, without the agent arguing. The others are a policy or permission check that trips, a misunderstanding that repeats, low confidence twice in a row, and a sensitive or vulnerable case. The model can raise a flag, but code decides whether a handoff fires and where it goes.
"Warm" means the context arrives before the person speaks. We send a packet with the call and show it to the receiving agent as the line connects. A person who can greet the caller by name and say what has already been done saves the call. A person who asks "how can I help you today?" undoes most of what the agent achieved.
| Packet field | Why the receiving person needs it |
|---|---|
| Caller identity and how it was verified | So they do not repeat verification, or know they must |
| The request in one sentence | So they can confirm it instead of asking for it |
| Actions completed, with tool results | So nothing is done twice |
| Actions pending or refused, with the reason | So they know what is still open and what was blocked |
| Why the handoff fired | So they know whether the caller is frustrated, confused or at risk |
| Links to transcript and recording | So they can check a detail without asking again |
| Language of the call | So the right person picks up |
Plan for the case where nobody is free. A transfer to an unstaffed queue is worse than no transfer at all. The agent should say that nobody is available, offer a callback, and open a ticket carrying the same packet, which is the pattern Yuma describes as email capture with the recording and transcript attached.
How do you confirm irreversible actions by voice?
An irreversible action by voice needs a confirmation step that a screen does not. Code reads back the exact values, the caller says yes, and code runs the call once. The model proposes the action and never both proposes and confirms it, because a model summarising its own tool call is the step most likely to be wrong.
This figure is a framework. It shows the order we use and the failure routes, and it describes no specific product.
Read back from the tool arguments, not from the model's words. If the model drafts a refund of an amount to an account, code takes the amount, the date and the last digits of the account out of the arguments it is about to send, and speaks those. A model that says "I'll refund the full amount" while the arguments hold a partial amount is the error this step exists to catch.
Speak values in the form a caller can check. "Forty-two pounds fifty to the card ending 1234" can be confirmed. "Your refund of the eligible amount" cannot. Dates should include the day of the week where it helps, and a reference number should be read in short groups the caller can write down.
A yes has to be explicit. Silence, a hesitation sound or "I suppose" is not consent to cancel an order. We ask once more when the answer is unclear, then offer a person. Code records the confirmation against the exact values read back, which also becomes the audit record for the action. Yuma describes the same rule for its agent, with each change confirmed verbally before it is applied.
The call itself runs once, with an idempotency key, so a retry after a timeout cannot refund twice. When a tool errors or times out, the agent says so, and code checks the target system's state before anything is retried. Our guide on why AI agents report unfinished work as done covers expected-state checks and retry safety in more depth. Voice raises the stakes, because the caller will hang up believing whatever the agent said last.
Which actions need this at all is a permissions question rather than a voice question. Our guide on scoping AI agent permissions covers how to bind each action to a principal, a task and an expected state. How to roll out AI agent autonomy in five levels covers how an action earns the right to run with a confirmation, and later without one. On AVOX we put required disclosures in the tool contract itself, so a payment could not execute unless its disclosure had been given, whatever the prompt said.
Where does Jev fit in a voice agent?
Jev is optional in a voice agent, and it earns its place at the handoff and escalation decisions rather than in the conversation. TypeSafe built it as a decision model that answers typed questions with calibrated probabilities and never generates text, and our guide on introducing Jev into agentic workflows sets it out in full.
The useful question on a call is whether this caller needs a person now. Asked as a yes-or-no question over the recent turns, the caller's stated request and the agent's proposed next step, it gives a probability that code can set a threshold against. A generative model asked the same question returns prose and no measure of doubt, and it can answer differently on the next call.
Keep the check off the critical path. Run it while the agent's reply is playing, or between the caller's end of turn and the start of synthesis if it fits the budget. Never let it replace the hard triggers. A caller asking for a person gets a person whatever Jev scores, and a failed permission check escalates in code without asking anyone.
Start Jev in shadow, logging its score beside what happened, as the Jev guide describes for tool gates. After enough calls the logs show whether a high score came before handoffs that people judged right. They also show how often the agent carried on past a point where a person would have helped. Those two numbers set the first threshold, and they come from your calls rather than from ours.
What should a voice agent log?
Log one structured record per turn, linked into one record per call. Audio alone is not a log, because nobody will listen to a thousand calls to find the one where a refund went wrong. Our guide to building an audit trail for AI decisions sets out the decision record in general; voice adds a few fields.
| Record | What it holds | Why |
|---|---|---|
| Turn | Transcript, end-of-turn decision, interruption and what was trimmed | Explains why the agent spoke when it did |
| Tool call | Arguments, read-back text, confirmation, result, idempotency key | Proves what was agreed and what ran |
| Handoff | Trigger, destination, the packet as sent | Shows the person got the context |
| Timing | Per-stage time from end of speech to first audio | Finds the stage that spent the budget |
| Versions | Model, prompt, voice, detector settings | Makes a call reproducible later |
| Recording | A link, with retention set by your data policy | Settles a dispute the transcript cannot |
Keep personal data out of the fields that feed analytics. A transcript can carry a card number or a health detail, and a call record copied into a dashboard spreads it. We redact at write time for the analytics copy and keep the full recording under the retention and access rules the business already applies to call recordings.
The record also has to survive the vendor. If a managed pipeline holds the transcript and the timing, export them into your own store at the end of each call. A dispute months later cannot depend on a vendor's log retention.
How do you test and release a voice agent?
Test with recorded audio from real calls, replayed through each configuration you are considering, and release by share of calls with a way back. Scripted test calls miss the noise, accents, interruptions and code-switching that break production agents. A test set built from your own callers catches them.
Replay testing makes changes comparable. The same recorded audio runs through the old and new configuration, and the transcripts, turn decisions, tool calls and replies are compared side by side. Microsoft lists a Voice Live evaluation harness in preview that does this against its own pipeline. On a cascade, the same replay can run in CI, with the eval approach in how to build LLM evals applied to the text at each stage.
Transcription deserves its own check, on the words that matter. A transcript can be almost entirely right and still get the one policy number or surname wrong, and that error drives the wrong tool call. We score names, numbers, dates and addresses separately from the rest of the words. AssemblyAI's stack guidance recommends the same focus on entity errors, and suggests forking live audio to a second provider as a shadow comparison before switching.
Release by share of calls. Route a small share of calls to the agent, with the rest going to people as before, and widen the share as the logs support it. Yuma describes its rollout the same way, by percentage of calls. Keep the handoff rate, the repeat-call rate and the confirmations that were refused in view per call type. When a class of call starts to go wrong, send that class back to people and leave the rest running.
The autonomy guide's stages apply here unchanged. Our guide to building production AI agents covers the decide, act, verify and stop loop that the voice path sits inside. Voice adds real-time failure and a caller who cannot scroll back, and neither changes the rule that an action earns its authority from evidence.
Common questions
What is the difference between a cascade and a speech-to-speech voice agent?#
A cascade runs speech to text, a language model and text to speech as separate stages, so every step produces text that code can read, check and log. A speech-to-speech model takes audio in and produces audio out from one model. It handles tone and interruption more naturally, and gives code less to inspect before the agent speaks or acts.
Should we use a managed voice agent API or build our own stack?#
Use a managed pipeline when launch time and a small integration count matter more than control of each stage, and when you do not have engineers to own turn detection and telephony. Build a cascade when the agent takes irreversible or regulated actions and you need to check text and tool arguments at every step. Many teams start managed and move one stage in-house when logs show it failing.
What latency should a production voice agent target?#
Set the budget from your callers, measured as the gap between the end of their speech and the first sound of the reply, and track the distribution per turn rather than the average. A line where callers interrupt often needs a tighter loop than one where a short spoken acknowledgement can cover a lookup. Tune end-of-turn detection first, because it usually costs more than the model.
How should a voice agent handle interruptions?#
Stop playback at once, trim the conversation history to what the caller actually heard, and decide whether the interruption was a backchannel or a new request. Interruption stops speech but does not undo a tool call already sent, so confirmation has to happen before any irreversible action runs.
What is a warm handoff for an AI voice agent?#
A warm handoff transfers the call to a person along with a context packet they see before they speak. The packet holds who the caller is and how they were verified, what they want, what the agent has done and refused, why the handoff fired, and links to the transcript and recording. The caller should not have to repeat anything.
When should a voice agent hand a call to a human?#
Whenever the caller asks, on the first request. Also when a policy or permission check fails, when the agent misunderstands repeatedly, when confidence stays low, and when the case is sensitive or the caller may be vulnerable. Code decides the trigger and the destination, and the model can only raise a flag.
How do you confirm a refund or cancellation by voice?#
Code reads back the exact values from the tool arguments, such as the amount, the date and the last digits of the account. The caller must give an explicit yes, and code then runs the call once with an idempotency key. Unclear answers get one more question and then an offer of a person, and the recorded confirmation becomes the audit record.
What should we log from every voice call?#
One structured record per turn, linked into a call record. It should hold the transcript, the end-of-turn decision, any interruption and trimmed text, tool arguments with read-back and confirmation, handoff triggers and packets, per-stage timing and pinned versions. Keep a link to the recording, and redact personal data in the copy that feeds analytics.
How do you test a voice agent before it takes real calls?#
Replay recorded audio from real calls through each configuration and compare transcripts, turn decisions, tool calls and replies side by side. Score names, numbers and dates separately from the rest of the transcript. Then release by share of calls, widening the share only as handoff and repeat-call rates hold.
Further reading
- How to build production AI agents. The decide, act, verify and stop loop the voice path sits inside.
- How to scope AI agent permissions. Binding each spoken action to a principal, task and expected state.
- How to roll out AI agent autonomy in five levels. How an action earns the right to run with, and later without, confirmation.
- How to build an audit trail for AI decisions. The decision record each call and tool action writes.
- Introducing Jev into agentic workflows. The shadow-then-gate sequence for an escalation check.
- Why AI agents report unfinished work as done. Expected-state checks and retry safety behind "your refund is done".
- AVOX, the insurance voice agent, and its engineering write-up. Where the latency budget went, and compliance in the tool contract.
- Voice agents, speech to text and text to speech. Glossary entries for the three stages of a cascade.
- AssemblyAI, Using the Voice Agent API alongside an existing voice stack and The voice AI stack for building agents in 2026, 8 September 2026. The managed and bring-your-own lanes, and shadow testing by forking audio.
- Yuma AI, Yuma launches AI voice agent that completes support actions during live phone calls, 22 September 2026. Handoff-first design, verbal confirmation and gradual rollout.
- Voiso, AI Agents documentation. Warm handoff to a queue, agent or number with conversation context.
- Microsoft, Foundry updates for July and August 2026 and Voice Live release notes. Hosted agents with Voice Live, auto-truncation and the evaluation harness preview.