Realtime Voice Agents

A realtime voice agent is software you talk to and that talks back in a natural back-and-forth, not a press-to-talk exchange. Phone support bots, voice assistants, and AI tutors all qualify. What separates a good one from a frustrating one is rarely the quality of any single model. It is orchestration: knowing when you have finished speaking, answering fast enough to feel present, and stopping gracefully when you cut it off. Humans leave a gap of roughly 200 to 500 milliseconds between conversational turns, with the median near 200 ms. Every design decision in a voice agent is downstream of trying to hit that number with a stack of models in the loop.

Pipeline versus speech-to-speech#

There are two architectures. The classic cascading pipeline chains three models: speech-to-text transcribes the user, an LLM decides what to say, and text-to-speech voices the reply. Its virtues are control and observability: you get a transcript for free (which feeds logging, analytics, and guardrails), you can swap any stage independently, and you can put the strongest available text model in the middle. Its vice is that latency compounds, and naive implementations that wait for each stage to finish before starting the next land at 1.5 seconds or worse.

The alternative is a native speech-to-speech model: one model takes audio in and emits audio out, with no intermediate text handoff. OpenAI's Realtime API is the flagship example; its gpt-realtime-2 model (May 2026) carries a 128,000-token context window, adjustable reasoning effort, and connects over WebRTC, WebSocket, or SIP for telephony. Google's Gemini Live API does the same with native audio Gemini models, around 70 languages, and built-in barge-in and tool support. These models start speaking in roughly 300 to 600 ms, and because they hear audio rather than a transcript, they pick up tone, hesitation, and laughter that a pipeline flattens away. The trade is opacity: no clean transcript boundary to inspect or filter, less control over each stage, and historically weaker reasoning than frontier text models, a gap that is closing fast. Production teams in 2026 still run plenty of pipelines for the control, and increasingly route simple conversational products to speech-to-speech.

Turn detection: knowing when you are done#

The agent must decide the moment your turn ends. The cheap tool is voice activity detection (VAD), a small model that classifies audio frames as speech or silence. But silence is a terrible proxy for "finished": wait only 300 ms and you will interrupt someone mid-thought ("my account number is... uh..."); wait a safe 800 ms and every reply feels sluggish, which is how many agents end up at 1.5 seconds of dead air.

The fix is semantic turn detection: pair VAD with a lightweight classifier that reads the live transcript and judges whether the utterance is complete. A finished sentence like "what is my balance?" can trigger after about 200 ms of silence, while a trailing "and also..." extends the wait. LiveKit ships an open turn-detector model that works this way, and Deepgram's Flux goes further by folding turn detection into the speech recognizer itself, claiming 200 to 600 ms lower response latency with about 30% fewer false interruptions than VAD-plus-endpointing stacks. OpenAI's Realtime API exposes the same idea as semantic VAD, a server-side option.

Barge-in: being interruptible#

Barge-in is what happens when the user starts talking while the agent is mid-sentence. A polite human stops; an agent has to do four things at once: halt audio playback, flush the queued TTS audio it had already synthesized, cancel the in-flight LLM generation, and truncate the conversation history to what the user actually heard, so the agent does not believe it said things that never played. Skip the truncation and the next turn is built on a false history.

Two details make this hard. Echo cancellation must be solid, or the agent hears its own voice and interrupts itself. And not every user sound is an interruption: backchannels like "mm-hmm" and "right" are encouragement, not a request to stop, so good stacks classify them rather than killing playback on any detected speech. Even well-built systems keep talking for 200 to 400 ms after the user starts speaking; mishandled barge-in is the single most common way a voice agent feels broken.

The latency budget, stage by stage#

End-to-end latency is measured from the moment the user stops speaking to the first audio of the reply. Walk the pipeline with realistic mid-2026 numbers. Turn detection spends its silence wait, about 200 ms with a semantic detector, 500 to 800 with naive VAD. STT costs little at the boundary because streaming recognizers transcribe while the user is still talking, so finalizing the transcript adds maybe 50 to 200 ms. The LLM's time to first token is usually the dominant slice, 200 to 500 ms for a fast model before the first sentence is ready, and any tool calling round trip stacks on top. TTS adds 40 to 200 ms of time-to-first-audio, and transport adds 50 to 100 ms more, with telephony hops costing extra.

Summed naively that is well over a second, which is why streaming overlap is non-negotiable: STT runs during speech, the LLM streams tokens, and TTS starts on the first complete sentence rather than the full reply. A well-built pipeline lands around 500 to 800 ms; speech-to-speech APIs report first audio in roughly 300 to 600 ms. Either way, the engineering work is guarding the budget, because every retry, network hop, and oversized system prompt quietly spends it. A useful trick when the budget blows out: have the agent emit a short acknowledgment ("let me check that") while slower tool calls run.

Practical takeaways#

Treat a voice agent as a latency and turn-taking system that happens to contain models. Pick the pipeline when you need observability, guardrails on text, or the strongest reasoning model; pick speech-to-speech when conversational feel and simplicity dominate. Use semantic turn detection rather than raw silence thresholds, and test barge-in explicitly, including the history-truncation case and backchannel words. Measure end-to-end latency from end of user speech to first audio on real networks, budget each stage, and keep the total under a second, because users will forgive a wrong answer faster than a long silence.