To build a voice AI agent, pick the architecture by what you need to see and time inside each turn, and keep every action that moves money or changes a record in your own code. A voice AI agent is software that holds a spoken conversation with a caller, on a phone line or in an app, and acts on what they ask through tools.
Key facts
Definition
A voice AI agent holds a spoken conversation and acts through tools, and it is built as a cascade, a speech-to-speech model, or a managed pipeline, chosen by how much of each turn the team needs to see and time.
How Bigcircle does it
- Bigcircle builds the conversation loop and the tool layer as separate layers, so a payment or cancellation is checked in code whatever runs the audio.
- Bigcircle times every stage of a turn; on AVOX Realty that showed the extra Hindi and Hinglish time sat in transcription, not reasoning.
- Bigcircle put required disclosures in AVOX's tool contract, so a payment could not run until its disclosure had been given.
Comparison
A managed pipeline hands the audio path to a vendor, while a cascade leaves text and timing at every stage where code can check it.
Key number
On AVOX, Bigcircle's insurance voice agent, end-to-end response went from 2.5 to 3 seconds down to around 900 milliseconds in internal testing.
There are three ways to build one. A cascade runs speech to text, a language model and text to speech as separate stages. A speech-to-speech model hears the caller and answers in one model, with no text stage between. A managed pipeline is either of those, run by a vendor behind one connection.
In all three, the actions that move money or change a record sit behind typed tools that check permissions and required wording before anything runs. A vendor can run the audio for you, but it does not know which of your operations need a disclosure first or a person to approve them.
| Criterion | Cascade | Speech to speech | Managed pipeline |
|---|---|---|---|
| Stages you run | All of them | The model call and tools | Tools and policy |
| Text at each step | Yes, by construction | A side transcript from a separate model | What the vendor exposes |
| Swap one component | Yes | No | Only inside the vendor's options |
| Turn-taking | Yours to build and tune | Mostly inside the model | The vendor's, with settings |
| When it fits | Regulated actions, strict wording, more than one language | Open conversation where tone matters most | A first launch, or a team without voice engineers |
| Typical failure | Latency added at each hand-off between stages | An action taken on audio nobody can inspect | A limit you cannot see past when a call fails |
| How Bigcircle handles this | Our two bilingual voice agents ran as separate stages, each timed, which is how we found where the Hindi delay sat. | The typed tool layer stays outside the model, so a payment or a cancellation is checked in code whatever produced the audio. | The tool contract, its disclosure checks and the call record stay on your side of the vendor boundary, the same split between conversation loop and tool layer that AVOX used. |
A voice agent fails in real time, in front of the caller, with no chance to edit. A chat agent that pauses for two seconds looks like it is thinking. A voice agent that pauses for two seconds sounds broken, and the caller starts talking again just as the reply begins, so both sides talk over each other and the turn is lost.
A demo tests the wrong things, because a demo call is quiet, scripted and short, with a caller who waits politely for each answer. Real callers talk over the agent, switch language mid-sentence, call from cars and kitchens, and ask to change something they did an hour ago. Fluency in the demo tells you little about any of that, and nothing about what happens when the agent must stop and hand over.
This guide comes out of two voice agents we built on one runtime. AVOX handled insurance calls and AVOX Realty handled property search, both in English, Hindi and the mix of the two. Both were real client projects, built and tested end to end and not yet in production, so the numbers below come from internal testing.
A third build, RepliCoach, is our own sales-practice agent on the same runtime, with no client.
The guide sets out the three architectures and how to choose between them, then where the latency goes, turn-taking, the handoff, and confirming an irreversible action by voice. It closes with what to log and how to test and release.
What are the three ways to build a voice AI agent?
A voice agent turns caller audio into a reply and, often, into an action in another system. The three architectures differ in how many stages that path has and who runs each one. That decides what you can inspect, what you can tune, and what you can log when a call goes wrong.
This figure is a framework. It compares architecture shapes and says nothing about any vendor's latency, accuracy or price.
A cascade chains three models: streaming speech to text, a language model working on text, and streaming text to speech, with a turn detector deciding when the caller has finished. Every boundary between stages produces text you can read, log and check. AVOX and AVOX Realty were built this way, as turn detection, transcription, reasoning, a tool call and synthesis, each one timed on its own.
You can swap one stage without touching the others, and a required disclosure can be checked as a string before it is spoken.
A speech-to-speech model takes audio in and produces audio out from one model, calling your tools along the way. It handles interruptions and tone more naturally, because nothing is converted to text and back between turns. The cost is visibility.
Do not build compliance checks on a speech-to-speech model's transcript, because it is a copy. In OpenAI's Realtime API it comes from a separate transcription model, which OpenAI's own cookbook warns can disagree with what the response model heard.
A managed pipeline hands the audio path to a vendor behind one connection and one bill, and underneath it can be either shape. Your code supplies the prompt, the tools and the policy around them, and sees only what the vendor chooses to expose about each stage. That is the trade: you skip building turn detection and telephony, and when a turn goes wrong you can see less of why.
How do you choose between them?
Ask three questions, in this order: what delay will these callers tolerate, what must be checked or logged at each stage, and who will fix turn-taking when it goes wrong? Demo fluency is not on the list, because every architecture sounds good on a quiet, scripted call.
The tolerated delay depends on the use case, and it is not always looser where the stakes are lower. On AVOX Realty we found the tolerance tighter than on insurance. An insurance caller who waits a beat assumes the system is checking their policy, while a property buyer who waits a beat assumes they are talking to a bot. Measure the gap from the end of the caller's speech to the first sound of the reply, on your callers, before you decide how much of it each stage may spend.
The check question is usually settled by what the agent can do. If it reads out a renewal date, a speech-to-speech model with a read-only tool is often fine. If it moves money, cancels an order or changes a record under regulation, you want text at each step.
On AVOX we built speed and strictness as separate layers: the conversation loop was tuned for latency, and the tool layer for correctness. Every insurance operation went through a typed tool that checked the request against a fixed format and checked the caller's permissions before it ran.
Ownership is the question teams skip, because a managed pipeline moves turn detection, noise handling and telephony to a vendor. That saves months of work and also moves the fix for a dropped turn onto someone else's roadmap. Accept that on a first launch with no regulated actions, and not where a wrong turn decision can start a payment.
Do not rule out a cascade on latency. Our cascade on AVOX reached around 900 milliseconds, against the 1.6 seconds that AssemblyAI's stack guide of 30 September 2026 gives for a sequential cascade when it argues for its own managed API. The difference is whether each stage is timed and tuned, which is the next section.
| If the use case has | Start with | Because |
|---|---|---|
| Irreversible actions, required disclosures, audit | Cascade | Code can check text before it is spoken and arguments before a tool runs |
| Open conversation, coaching, low-stakes reads | Speech to speech | Turn-taking and tone matter more than inspecting each step |
| A first launch and no voice team | Managed pipeline | Telephony and turn detection are already built and run by someone else |
| Callers mixing languages in one sentence | Cascade, or a managed pipeline that exports per-stage timing | Turn detection and transcription fail first on mixed-language audio, and you need to see which one |
Where does the latency budget go?
The budget is spent across every stage of a turn, and the stage that decides the most is the one teams tune last. The usual assumption is that the language model is the slow part. On AVOX the component that breaks first, and the one that received the most tuning, is the decision about when the caller has finished speaking.
This figure is a framework showing the order of stages in one cascade turn. It carries no timings, because those depend on your models, network and telephony.
On AVOX, end-to-end response went from 2.5 to 3 seconds down to around 900 milliseconds on noisy bilingual calls. Turn detection got the most work, because a saving in synthesis is invisible next to a detector that waits too long to decide the caller has stopped.
Two other changes mattered on AVOX. Policy and claim context was loaded when the call opened rather than fetched mid-turn, which took the largest variable cost off the path of every turn. Every stage was timed separately, because "the agent felt slow" cannot be checked, and per-stage time tells you whether to look at detection, transcription, reasoning, a tool or synthesis.
Per-stage timing paid off again on AVOX Realty, where replies came back in under 1.2 seconds in English and under 1.3 seconds in Hindi and Hinglish. The timing showed that the extra time on the Hindi and Hinglish paths was spent in transcription, not in reasoning.
In a cascade that points at one stage you can tune or replace. A speech-to-speech model has no separate transcription stage to time, so the same slowdown would show up only as a slower model.
Tool calls are the other place the budget goes, because a lookup that takes a second on a web page is a second of silence on a call. A spoken "let me check that" while a slow tool runs fills the gap without claiming anything, but "your refund is done" before the tool returns is a claim nobody has checked.
Keep the per-turn gap as a distribution rather than an average, because a good mean with a long tail still leaves some callers in a three-second silence. The latency metrics page covers the terms for each part.
How should turn-taking and barge-in work?
Turn-taking decides when the caller has finished and the agent may speak. Barge-in decides what happens when the caller speaks over the agent. On AVOX we treated both as state problems more than audio problems: an interruption has to drop the reply in progress and continue the conversation without starting the turn again.
A silence threshold is the simplest end-of-turn detector: wait for a gap of a set length and assume the caller is done. That works on quiet, fluent speech, but on a call where the caller pauses to find a policy number, the pause is not an ending.
Property buyers on AVOX Realty switched language inside a clause ("2BHK chahiye, but near the metro"), and the pauses fell in places neither language would put them. A detector tuned on clean switching between two languages lost accuracy there. We considered a separate flow per language and rejected it, because there was no clean point at which to switch.
As of October 2026 the main voice APIs offer both a silence-based detector and a semantic one, which judges from the words so far whether the caller has finished. Whichever you use, test it on your own callers' audio, because a detector tuned on read speech tells you little about a caller reading a claim number from a letter.
Barge-in needs three steps, and missing one breaks the conversation. Playback stops at once, so the agent does not keep talking over the caller. The conversation history is trimmed to what the caller actually heard, because otherwise the model believes it finished a sentence the caller cut off. Then the interruption is classed as a backchannel or as a new turn.
On a bilingual line, check the backchannel handling in both languages. A filler-word list that ignores "mm" and "yeah" while the agent speaks is usually English, and a Hindi acknowledgement the list does not know becomes a false barge-in.
History trimming is the step most often missed, and it shows up on the next turn. If the agent was reading three delivery options and the caller cut in after the first, the model's history still holds all three. On the next turn it may refer to one the caller never heard. Check that trimming is switched on in a managed API, where it can be off by default, and build it by hand in a cascade.
Sometimes the agent has to interrupt the caller. RepliCoach plays a difficult buyer and pushes back on the sales rep mid-sentence, so it decides when to cut in before the rep has finished. Getting that timing wrong was the failure users noticed: too eager and it talks over the rep, too slow and the objection arrives as a comment. The timing had to be tuned per persona.
Barge-in stops speech, and it does not stop an action. If a tool call was already sent when the caller interrupted, it is running, and the interruption changes nothing about its effect. Confirmation therefore has to come before an irreversible call, which the section after the handoff covers.
Why design the handoff first?
A production voice agent will reach calls it should not handle, and the handoff decides what the caller thinks of the whole system. Design it before the happy path, because otherwise it exists only as a last-minute transfer to a queue, with the caller repeating everything to a person who knows nothing.
On AVOX, a tool request that failed validation, a permission check or a disclosure gate did not fall back to something close. It escalated: an agent that cannot complete a payment hands off rather than improvising an adjacent action. The escalation carried the dialogue and the actions already taken, because a handoff that makes the caller repeat everything turns a partial success into a worse call than no agent at all.
This figure is a framework. The triggers and packet fields are a design checklist, not a vendor's feature list, and the figure reports no measured result.
The triggers belong in code, not in the prompt. A caller asking for a person is the obvious one, and it should work on the first request, without the agent arguing. The others are a policy or permission check that trips, a misunderstanding that repeats, low confidence twice in a row, and a sensitive or vulnerable case. The model can raise a flag, but code decides whether a handoff fires and where it goes.
"Warm" means the context arrives before the person speaks. Send a packet with the call and show it to the receiving agent as the line connects. A person who can greet the caller by name and say what has already been done saves the call. A person who asks "how can I help you today?" undoes most of what the agent achieved.
| Packet field | Why the receiving person needs it |
|---|---|
| Caller identity and how it was verified | So they do not repeat verification, or know they must |
| The request in one sentence | So they can confirm it instead of asking for it |
| Actions completed, with tool results | So nothing is done twice |
| Actions pending or refused, with the reason | So they know what is still open and what was blocked |
| Why the handoff fired | So they know whether the caller is frustrated, confused or at risk |
| Links to transcript and recording | So they can check a detail without asking again |
| Language of the call | So the right person picks up |
Plan for the case where nobody is free, because a transfer to an unstaffed queue is worse than no transfer at all. The agent should say that nobody is available, offer a callback, and open a ticket carrying the same packet with the recording and transcript attached.
How do you confirm irreversible actions by voice?
An irreversible action by voice needs a confirmation step that a screen does not. On a screen the user reads a proposed refund before pressing a button. On a call the caller hears it once, so code reads back the exact values, the caller says yes, and code runs the call once. The model proposes the action and never also confirms it, because a model summarising its own tool call can get the summary wrong.
This figure is a framework. It shows the order of steps and the failure routes, and it describes no specific product.
Read back from the tool arguments, not from the model's words. If the model drafts a refund of an amount to an account, code takes the amount, the date and the last digits of the account from the arguments it is about to send. Those are what the agent speaks. A model that says "I'll refund the full amount" while the arguments hold a partial amount is the error this step catches.
Speak values in a form the caller can check on the spot. "Forty-two pounds fifty to the card ending 1234" can be confirmed, and "your refund of the eligible amount" cannot. Give the day of the week with a date where it helps, and read a reference number in short groups the caller can write down.
A yes has to be explicit, and silence, a hesitation sound or "I suppose" is not consent to cancel an order. Ask once more when the answer is unclear, then offer a person. Record the confirmation against the exact values read back, so it doubles as the audit record for the action.
Run the call once, with an idempotency key, so a retry after a timeout cannot refund twice. When a tool errors or times out, the agent says so, and code checks the target system's state before anything is retried. Our guide to AI agent verification covers those state checks in more depth. They matter more on a call, because the caller will hang up believing whatever the agent said last.
Some actions also need words said before they can run. On AVOX the required disclosures sat in the tool contract rather than in the prompt, so a payment could not execute until its disclosure had been given. "Will the model always make the disclosure?" has no answer you can test. "Can this operation run without the disclosure?" does, and the answer was no.
Which actions need any of this is a permissions question. Our guides to AI agent permissions and AI agent governance cover how to tie each action to the person it is done for and the task it belongs to. They also cover how it earns the right to run with a confirmation and later without one.
What carries over to a second voice agent?
The conversation loop carries over and the tools do not, if the line between them is drawn in the right place. AVOX Realty reused the AVOX real-time loop intact: turn detection and noise handling, interruption, latency timing, session and turn logging, and the split between voice workers and the API.
We rebuilt the domain layer: typed tools for locality search, amenities, commute time and the CRM, a data layer for listings and buyer preferences, and loaders that ground answers in current inventory.
That only works if the loop knows nothing about the business and the tools know nothing about audio. Where a domain idea leaks into the loop, such as the loop knowing what a policy or a locality is, the port stops being a configuration change and becomes a rewrite.
RepliCoach is the third domain on the same runtime, and it cost a tool layer and a scoring rubric rather than a build. A managed pipeline can keep the same line, as long as your tools sit behind your own contract rather than inside the vendor's agent configuration.
Where does Jev fit in a voice agent?
Jev is optional in a voice agent, and it belongs at the handoff decision rather than in the conversation. TypeSafe describes it as a decision model that gives up text generation and returns typed values with calibrated probabilities. Our guide to introducing TypeSafe's Jev into agentic workflows sets it out in full.
The useful question on a call is whether this caller needs a person now. Asked as a yes-or-no question over the recent turns, the caller's request and the agent's proposed next step, it returns a probability that code can set a threshold against. A generative model asked the same question returns prose and no measure of doubt.
Keep the check off the per-turn path. TypeSafe's launch post of 15 September 2026 gives 70 to 500 milliseconds end to end. Against a turn of around a second that is too large a share to spend inline, so run it while the agent's reply is playing. Never let it replace the hard triggers: a caller asking for a person gets a person whatever Jev scores, and a failed permission check escalates in code without asking anyone.
Start it in shadow, logging its score beside what happened, as the Jev guide describes for tool gates. The logs then show whether high scores came before handoffs people judged right, and how often the agent carried on past a point where a person would have helped. Those two counts set the first threshold, and only your own calls can supply them.
What should a voice agent log?
Log one structured record per turn, linked into one record per call. Audio alone is not a log, because nobody will listen to a thousand calls to find the one where a refund went wrong.
On AVOX each call wrote turns, transcripts, the tools invoked with their arguments and results, the disclosures made, escalations and the outcome. QA, claim checks and compliance review all read that same record. Our guide to an AI audit trail sets out the decision record in general, and voice adds a few fields.
| Record | What it holds | Why |
|---|---|---|
| Turn | Transcript, end-of-turn decision, interruption and what was trimmed | Explains why the agent spoke when it did |
| Tool call | Arguments, read-back text, confirmation, result, idempotency key | Proves what was agreed and what ran |
| Handoff | Trigger, destination, the packet as sent | Shows the person got the context |
| Timing | Per-stage time from end of speech to first audio | Finds the stage that spent the budget |
| Versions | Model, prompt, voice, detector settings | Makes a call reproducible later |
| Recording | A link, with retention set by your data policy | Settles a dispute the transcript cannot |
Keep personal data out of the fields that feed analytics. A transcript can carry a card number or a health detail, and a call record copied into a dashboard spreads it. Redact the analytics copy at write time, and keep the full recording under the retention and access rules the business already applies to call recordings.
The record also has to outlive your contract with the vendor. If a managed pipeline holds the transcript and the timing, export them into your own store at the end of each call. A dispute months later cannot depend on a vendor's log retention.
How do you test and release a voice agent?
Test with recorded audio from real calls, replayed through each configuration you are considering, and release by share of calls with a way back. Scripted test calls miss the noise, accents, interruptions and language switching that break production agents, and a test set built from your own callers catches them.
Replay makes two configurations comparable, because the same audio runs through both and you compare transcripts, turn decisions, tool calls and replies call by call. Microsoft's evaluation harness for its voice API does this with pre-recorded audio. Its documentation, updated in September 2026, says it scores conversational quality from the transcribed text and does not measure word error rate or speech quality.
It tells you whether the words were right, not whether the turn worked. Our guide to voice agent testing covers how to score the audio as well, and the one on LLM evaluation covers the text side.
Transcription deserves its own check on the words that matter. A transcript can be almost entirely right and still get the one policy number or surname wrong, and that error drives the wrong tool call. Score names, numbers, dates and addresses separately from the rest.
Compare transcription providers on that entity score, not on overall accuracy. Before switching, fork the live audio to the new model in shadow, so it transcribes the same calls as the current one with nothing it produces reaching the caller.
Release by share of calls: route a small share to the agent, with the rest going to people as before, and widen it as the logs support.
Watch the handoff rate, the repeat-call rate and the refused confirmations per call type. When one class of call starts to go wrong, send that class back to people and leave the rest running. Our guide on how to build an AI agent covers the decide, act, verify and stop loop that the voice path sits inside.
Common questions
What is an AI voice agent?#
An AI voice agent is software that holds a spoken conversation with a caller, on a phone line or in an app, and acts on what they ask through tools, such as looking up a claim or searching current property listings. It is built as a cascade of speech to text, a language model and text to speech, as one speech-to-speech model, or as a managed pipeline a vendor runs.
What is the difference between a cascade and a speech-to-speech voice agent?#
A cascade runs speech to text, a language model and text to speech as separate stages, so every step produces text that code can read, check, time and log. A speech-to-speech model takes audio in and produces audio out from one model. It handles tone and interruption more naturally, and its transcript is a side output, so code has less to inspect before the agent speaks or acts.
Should we use a managed voice agent API or build our own stack?#
Use a managed pipeline when launch time and a small integration count matter more than control of each stage, and when nobody on the team will own turn detection and telephony. Build a cascade when the agent takes irreversible or regulated actions and you need to check text and tool arguments at every step. In either case keep the tools, their checks and the call record in your own code.
What latency should a production voice agent target?#
Set the target from your callers, measured as the gap between the end of their speech and the first sound of the reply, and track it per turn as a distribution. On our insurance agent the gap went from 2.5 to 3 seconds to around 900 milliseconds in internal testing, and turn detection took most of the work. Tolerance also differs by use case: property buyers on our second agent were less patient than insurance callers.
How should a voice agent handle interruptions?#
Stop playback at once, trim the conversation history to what the caller actually heard, and decide whether the interruption was a backchannel or a new request. Interruption stops speech but does not undo a tool call already sent, so confirmation has to happen before any irreversible action runs.
What is a warm handoff for an AI voice agent?#
A warm handoff transfers the call to a person along with a context packet they see before they speak. The packet holds who the caller is and how they were verified, what they want, what the agent has done and refused, why the handoff fired, and links to the transcript and recording. The caller should not have to repeat anything.
When should a voice agent hand a call to a human?#
Whenever the caller asks, on the first request. Also when a policy or permission check fails, when the agent misunderstands repeatedly, when confidence stays low, and when the case is sensitive or the caller may be vulnerable. Code decides the trigger and the destination, and the model can only raise a flag.
How do you confirm a refund or cancellation by voice?#
Code reads back the exact values from the tool arguments, such as the amount, the date and the last digits of the account. The caller must give an explicit yes, and code then runs the call once with an idempotency key. An unclear answer gets one more question and then an offer of a person, and the recorded confirmation becomes the audit record.
What should we log from every voice call?#
One structured record per turn, linked into a call record. It should hold the transcript, the end-of-turn decision, any interruption and trimmed text, tool arguments with read-back and confirmation, handoff triggers and packets, per-stage timing and pinned versions. Keep a link to the recording, and redact personal data in the copy that feeds analytics.
How do you test a voice agent before it takes real calls?#
Replay recorded audio from real calls through each configuration and compare transcripts, turn decisions, tool calls and replies call by call. Score names, numbers and dates separately from the rest of the transcript. Then release by share of calls, widening the share only while handoff and repeat-call rates hold.
Further reading
- AVOX, the insurance voice agent, and its engineering write-up. Where the latency budget went, and compliance in the tool contract.
- AVOX Realty, the real-estate voice agent. The same runtime in a second domain, and where the Hindi delay sat.
- RepliCoach, the sales roleplay agent. An agent that has to interrupt, on the same runtime.
- How to build an AI agent. The decide, act, verify and stop loop the voice path sits inside.
- AI agent permissions. Binding each spoken action to a principal, task and expected state.
- AI audit trail. The decision record each call and tool action writes.
- Introducing TypeSafe's Jev into agentic workflows. The shadow-then-gate sequence for an escalation check.
- Voice agents, speech to text and text to speech. Glossary entries for the three stages of a cascade.
- Microsoft, How to evaluate Voice Live agents, updated September 2026. A replay harness that scores from the transcribed text.
- OpenAI cookbook, out-of-band transcription. Why the side transcript can differ from what the model heard.
- AssemblyAI, The voice AI stack for building agents in 2026, 30 September 2026. The vendor's cascade latency figure.
- TypeSafe, Introducing System One models and Jev, 15 September 2026. The launch post and the vendor's latency figure.