Realtime Voice Agents

A voice agent is software that holds a spoken conversation with a caller, on a phone line or in an app, and acts on what they ask through tools, such as looking up a claim or searching current listings. A realtime voice agent does it as a natural back-and-forth rather than press-to-talk. A frustrating voice agent rarely has a worse model than a good one. It decides badly when the caller has finished, answers after the pause already sounds like a dropped line, or keeps talking when the caller cuts in.

A voice agent fails in front of the caller, with no chance to edit. A chat agent that pauses for two seconds looks like it is thinking. A voice agent that pauses for two seconds sounds broken, and the caller starts talking again just as the reply begins.

Cascade, speech-to-speech or managed#

There are three ways to build one. A cascade chains speech-to-text, a language model and text-to-speech as separate stages, so every boundary produces text that code can read, check and time. A speech-to-speech model hears audio and answers in audio from one model. OpenAI's Realtime API (on gpt-realtime-2.1 as of October 2026) and Google's Gemini Live API, which lists 70 supported languages, both work this way and both call tools. A managed pipeline is either shape, run by a vendor behind one connection.

Choose by what the agent can do. If it reads out a renewal date, a speech-to-speech model with a read-only tool is often fine. If it moves money, cancels an order or changes a record under regulation, build a cascade, because code can check text before it is spoken and arguments before a tool runs. Do not build compliance checks on a speech-to-speech transcript: in OpenAI's Realtime API it comes from a separate transcription model, and OpenAI's own cookbook warns it can disagree with what the response model heard.

AVOX, the insurance agent, and AVOX Realty, the property agent, are cascades we built for clients on one runtime, in English, Hindi and the mix of the two.

Where the latency goes#

Measure the gap from the end of the caller's speech to the first sound of the reply, and keep it as a distribution per turn, because a good average with a long tail still leaves some callers in a three-second silence. Set the target from your own callers rather than from a universal number. On AVOX Realty we found property buyers less patient than insurance callers: an insurance caller who waits a beat assumes the system is checking their policy, and a buyer who waits assumes they are talking to a bot.

On AVOX, we measured end-to-end response falling from 2.5 to 3 seconds to around 900 milliseconds on noisy bilingual calls in testing. On AVOX Realty, replies came back in under 1.2 seconds in English and under 1.3 seconds in Hindi and Hinglish. Turn detection took most of the tuning, as the next section covers, and two other changes mattered. Each stage was timed separately, which on AVOX Realty showed that the extra time on the Hindi and Hinglish paths was spent in transcription rather than reasoning. And on AVOX the policy and claim context loaded when the call opened, instead of being fetched mid-turn.

Overlap the stages. Streaming recognition transcribes while the caller is still talking, the language model streams tokens, and synthesis starts on the first complete sentence. When a slow tool has to run, a spoken "let me check that" fills the gap without claiming anything. "Your refund is done" before the tool returns is a claim nobody has checked.

Turn detection breaks first#

The usual assumption is that the language model is the slow part. On AVOX, the decision about when the caller had finished was the component that broke first and took the most tuning. A silence threshold waits for a gap of a set length. Make it short and the agent cuts in while a caller pauses to find a policy number. Make it long and every reply arrives late.

A semantic detector judges from the words so far whether the caller has finished, so a complete question can end the turn after a brief pause while "and also..." holds it open. As of October 2026 the main voice APIs offer both kinds. Deepgram reports that its Flux model gives 200 to 600 ms lower response latency and about 30% fewer false interruptions than pipeline approaches. Test any detector on your own callers' audio. On AVOX Realty buyers switched language inside a clause ("2BHK chahiye, but near the metro"), and a detector tuned on clean switching between languages lost accuracy there. We considered a separate flow per language and rejected it, because there was no clean point at which to switch.

Barge-in and the tools behind it#

Barge-in is the caller speaking while the agent is still talking, and handling it takes three steps. Stop playback at once. Trim the conversation history to what the caller actually heard, or the model will refer next turn to an option the caller never heard. Then decide whether the sound was a backchannel or a new turn. On a bilingual line, check the backchannel list in both languages, since a Hindi acknowledgement the list does not know becomes a false barge-in.

Barge-in stops speech and does not stop an action. If a tool call was already sent, it is running. On AVOX the model's output was only ever a request to call a tool, and each tool validated its arguments, checked the caller's permissions, and enforced required disclosures before it ran. A tool request that failed any check did not fall back to something close. It escalated to a person, carrying the dialogue and the actions already taken, so the caller did not have to repeat anything.

Our guide on how to build a voice AI agent covers the handoff packet, reading back an irreversible action, and what to log per turn. Voice agent testing covers scoring the audio separately from the transcript, because a correct transcript can still be a call the caller could not use.