Speech-to-Text (ASR)

Speech to text for voice agents is the recognition step that turns a caller's audio into text the agent can reason over. Speech-to-text, also called automatic speech recognition or ASR, is the same technology behind dictation and meeting captions. A voice agent puts two extra demands on it: the text has to arrive while the person is still speaking, and the agent will act on it, so a wrong word becomes a wrong action rather than a typo in a transcript.

The usual comparison of engines ranks them on one accuracy number from a public test set. For a voice agent that number is the least useful of the three that matter. The other two are accuracy on your own callers' names, numbers and phrasing, and how much of your response time recognition spends.

Batch versus streaming#

Batch transcription takes a complete recording and returns the full transcript. Because the model hears the whole utterance, it can use later words to fix earlier ones, so batch accuracy beats streaming accuracy on the same engine. Deepgram's Nova-3 announcement reports a median word error rate of 5.26% on batch audio and 6.84% streaming. OpenAI's Whisper, released in 2022 under the MIT licence with 99 languages at launch and six model sizes from 39M to 1.55B parameters, was designed batch-first and is still the common choice for self-hosted recordings.

Streaming transcription consumes audio as the person speaks and emits partial guesses that update and then lock into final text. You will see "book a flite to" correct itself to "book a flight to" as more sound arrives. Voice agents need this mode, because waiting for the speaker to finish before transcription starts adds dead air to every turn. Whisper has no native streaming mode. Wrappers that re-run it over overlapping windows spend compute and still trail engines built to stream, so use a purpose-built streaming engine for anything live.

Word error rate, measured on your audio#

Accuracy is reported as word error rate (WER): substitutions plus insertions plus deletions, divided by the number of words actually spoken. If a patient says "please refill my prescription for metformin" and the transcript reads "please refill my prescription for met forming", that is one substitution and one insertion against six reference words, a WER of 33% on the sentence that matters most. WER counts every word the same, so a system can post 5% overall while missing exactly the drug names, product codes and customer names your application runs on.

Public test sets are mostly clean speech. Your traffic has accents, cross-talk, speakerphone echo, and terms the model never saw. Build a test set of 30 to 60 minutes of real recordings, have a person write the reference transcripts, score candidates with an open tool such as jiwer, and treat it like any other eval. Then score the words that matter separately from the average.

On the on-device meeting transcription app we built with a sales-software client's team, we measured the transcription error rate falling from about 18% to 10% across nine rounds of tuning, with the largest gains on filler words and industry terms. Those were the errors that used to break the automation downstream. Decoding there is biased toward sales and product vocabulary. Hosted engines offer the same lever: Deepgram's keyterm prompting accepts up to 100 terms per request, and OpenAI's transcription models accept a text prompt that steers spelling, except the diarize model. Try vocabulary biasing before any fine-tune.

The transcript has a contract#

A voice agent, a CRM sync or a summary reads the transcript as input, and filler words, stutters and invented phrases do not degrade that output gracefully; they break it. On the meeting transcription app, cleaning sits before the language model rather than after it, because cleaning once at the source is worth more than repairing the same noise in three consumers. That build runs a compact, quantized speech model on the laptop's CPU with no upload path at all, because cloud transcription was a disqualification for teams with data-residency obligations.

Code-switching is where recognition fails in ways a WER average hides. On the insurance and real-estate voice agents we built, callers switched between English and Hindi inside a clause, as in "2BHK chahiye, but near the metro", and a code-switch mid-clause produced transcript fragments grammatical in neither language. A per-language mode switch would have been the obvious design and the wrong one, so those agents run one path for English, Hindi and Hinglish.

Latency, measured per stage#

Recognition has two clocks. Time to partial is how fast any text appears, and time to final is how long until a word is locked and safe to act on. Deepgram's documentation, as of October 2026, describes Nova-3 streaming latency as under 300 ms. A voice agent adds a third: end-of-speech detection, deciding the caller has stopped so the agent may answer. Fire early and the agent interrupts; fire late and every reply opens with silence. On the insurance agent, turn detection broke first and received the most work of any component. Deepgram's Flux, launched in October 2025, is a recognition model built around that end-of-turn decision.

Instrument every stage, because "the agent felt slow" cannot be fixed. On the real-estate agent, per-stage timing showed the Hindi and Hinglish paths spending their extra time in transcription rather than in reasoning, and replies held under 1.2 seconds in English and 1.3 seconds in Hindi and Hinglish. On the meeting app, per-stage state reporting exists for the same reason: without it, every ticket becomes a model investigation when most are audio-routing problems. Budget the whole loop the way you would any latency metric: recognition, the language model's time to first token, tool calls and text-to-speech.

Choosing an engine in 2026#

Use a self-hosted model, Whisper or NVIDIA's Parakeet and Canary families, for batch work and for audio that cannot leave your infrastructure. For live calls, shortlist purpose-built streaming engines such as Deepgram's Nova-3 and Flux, AssemblyAI's streaming models, ElevenLabs' Scribe v2 Realtime and OpenAI's realtime transcription, then let your own recordings choose. AssemblyAI's Universal-2 reports up to a 24% improvement in proper-noun accuracy over its predecessor, a figure worth checking against your own names. Prices move often. As of October 2026, Nova-3 batch is $0.0043 per minute, its English streaming rate is on a promotion at $0.0048 per minute against a list rate of $0.0077, and AssemblyAI lists Universal-2 batch at $0.15 per hour.

As of October 2026, OpenAI's deprecations page schedules whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe and gpt-4o-transcribe-diarize to leave the API on 26 February 2027, with gpt-live-transcribe or gpt-transcribe listed as the replacements. A new integration should not start on those ids. GPT-4o deprecation: dates, replacement and how to migrate is the eval pass when the transcription model is the id that changes.