Speech-to-Text (ASR)

Speech-to-text, also called automatic speech recognition or ASR, converts spoken audio into text. It is the front door of every voice product: the dictation in your notes app, the captions on a video call, the first step a voice agent takes before it can think. OpenAI's Whisper made high-quality open ASR free in 2022 and is still everywhere, but the 2026 landscape splits cleanly along one axis: are you transcribing a finished recording, or a live conversation? Almost every other decision follows from that.

Batch versus streaming#

Batch transcription takes a complete audio file and returns the full transcript at once. Because the model sees the whole utterance, it can use later words to fix earlier ones, which is why batch accuracy beats streaming accuracy for the same vendor: Deepgram's Nova-3, for example, reports a median word error rate of 5.26% on batch audio but 6.84% on the same engine streaming. Whisper was designed batch-first and excels at it. A one-hour podcast sent to a batch endpoint typically transcribes in well under a minute and costs a fraction of a cent per audio minute (Nova-3 batch is $0.0043 per minute; AssemblyAI is around $0.15 per hour).

Streaming transcription instead consumes audio as the person speaks, emitting partial hypotheses that update live and then lock into finalized text. The partials are allowed to be wrong; you will see "book a flite to" correct itself to "book a flight to" once more acoustic context arrives. Live captioning, dictation, and voice agents all need this mode, because waiting for the speaker to finish before transcription even starts adds seconds of dead air. Whisper has no native streaming mode; community wrappers fake it by re-running overlapping windows, which burns compute and still lags purpose-built streaming engines like Deepgram, AssemblyAI's streaming models, or OpenAI's Realtime transcription endpoint.

WER, measured on your audio#

ASR accuracy is reported as Word Error Rate: substitutions plus insertions plus deletions, divided by the number of words actually spoken. If a patient says "please refill my prescription for metformin" and the transcript reads "please refill my prescription for met forming," that is one substitution and one insertion against six reference words, a 33% WER on the sentence that matters most. WER counts every word equally, so a system can post 5% overall while missing exactly the drug names, SKUs, and customer names your application lives on.

That is why the only WER that matters is measured on your audio. Vendor leaderboards are mostly clean, read-aloud English; your production traffic has Indian and Scottish accents, cross-talk, speakerphone echo, and domain jargon the model never saw. The spread is dramatic: one 2026 benchmark roundup puts gpt-4o-transcribe near 2.5% WER under favorable conditions while open Whisper large-v3-turbo lands around 15% on challenging real-world audio. Build a 30 to 60 minute test set from real recordings, have a human produce reference transcripts, and score candidates with an open tool like jiwer; treat it like any other eval. Most vendors also offer cheap fixes worth testing before fine-tuning anything: Deepgram's keyterm prompting accepts up to 100 domain terms per request, and OpenAI's transcribe models accept a text prompt that biases spelling toward your vocabulary.

Latency for realtime#

For live applications, latency sits beside accuracy as a first-class metric, and it has two distinct figures. Time-to-partial is how fast any text appears; time-to-final is how long until a word is locked and safe to act on. Purpose-built streaming engines finalize words in under 300 milliseconds, while bolted-on streaming can lag by seconds. For voice agents there is a third number that dominates the user experience: end-of-speech detection, how quickly the system decides you have stopped talking and the agent may answer. Get that wrong in one direction and the agent interrupts you; wrong in the other and every reply starts with an awkward second of silence. Models like Deepgram's Flux are now built specifically around fast end-of-turn detection. Budget the whole loop the way you would any latency metric: roughly 300 ms for ASR, the LLM's time-to-first-token, then text-to-speech synthesis, all of which must fit inside the approximately one second a human tolerates before a conversation feels broken.

Whisper and the 2026 alternatives#

Whisper remains the open-source default: MIT-licensed, 99 languages, six sizes from 39M to 1.55B parameters, ideal for self-hosting and offline batch work, with NVIDIA's Parakeet and Canary families as faster open alternatives. On the hosted side, OpenAI's API has moved past whisper-1 to the gpt-4o-transcribe family (including a mini tier and a diarize variant that labels speakers), which leads several independent accuracy tests. Deepgram's Nova-3 is the streaming workhorse with sub-300 ms latency and aggressive pricing at $0.0077 per streaming minute. AssemblyAI's Universal-2 pairs strong accuracy with the best transcript post-processing, claiming up to 24% better proper-noun handling, plus an analytics stack (summaries, sentiment, PII redaction). ElevenLabs' Scribe v2 targets multilingual realtime. The honest summary: self-host with Whisper or Parakeet, stream with Deepgram, mine transcripts with AssemblyAI, chase peak accuracy with gpt-4o-transcribe, then let your own test set overrule the defaults.

Practical takeaways#

Pick batch for recordings and purpose-built streaming for anything live; do not stretch Whisper into a streaming role it was never designed for. Measure WER on 30 to 60 minutes of your real audio, with your accents and your jargon, before trusting any leaderboard. Feed domain terms through keyterm or prompt biasing; it is the cheapest accuracy win available. For interactive apps, hold vendors to under 300 ms time-to-final and scrutinize end-of-speech detection, because turn-taking, not raw WER, is what makes a voice agent feel human.