Voice agent testing has to score the audio as well as the words, because an answer that is correct in the transcript can still fail the caller. Turn-taking, interruptions, whether the speech could be followed, and how long the silences ran all sit outside a text rubric. A call should pass only when the text checks and the audio checks both pass, and a change should be shown on the same recorded calls before it takes traffic.
Key facts
Definition
Voice agent testing scores the call, not only the transcript: turn-taking, interruptions, whether the speech could be followed, and how the silence felt.
How Bigcircle does it
- On RepliCoach, Bigcircle records turns, speakers and timestamps as structured events at the time of speaking, because rebuilding them from audio later was lossy and too slow.
- Bigcircle timed every stage of a turn on AVOX and AVOX Realty, so a slow call could be pinned to one stage instead of to the agent as a whole.
- The RepliCoach scoring rubric is versioned and kept separate from the model, so Bigcircle can change the model without moving the scale.
Comparison
A text rubric can pass an answer the caller still could not follow, which is why the audio checks sit beside the transcript rather than inside it.
Key number
On AnyTeam, in production, Bigcircle's tuning took word error rate from roughly 18% to 10% over nine rounds, by Bigcircle's own measurement.
The failure never shows up in the record most teams score. Take a claim-status call, as an illustration rather than a measured result. The transcript shows the correct status, the next step and no policy breach, so a judge reading it marks the call passed. On the recording, the caller waited through a silence, started a second sentence, and was talked over by the reply. The words were fine and the turn was lost.
Nothing in a transcript-only trace can turn that pass into a fail, because the overlap between the two speakers was never written down as a field the judge could read. The caller decides whether to phone again about the same request, and a green transcript score beside those repeat calls is the disagreement this guide is about.
This guide draws on four voice systems we built and what each of them measured. AVOX and AVOX Realty are bilingual phone agents for an insurer and a property business, built and tested end to end, not yet in production. RepliCoach is our own sales-practice agent, which scores every session. AnyTeam is an on-device meeting transcriber that is in production. The release gate this guide sets out for a live line is the design we recommend from those tests.
The guide starts with what a transcript judge cannot see, then separates content failures from experience failures and sets out the audio signals that tell them apart. It then covers how those signals sit next to text judges without being averaged, how a frozen set proves a fix, and what each voice architecture leaves you able to measure.
What do transcript judges miss?
Transcript judges miss the timing of the turn, the interruptions, and whether the caller could follow the speech. They read the words after the call and score whether those words were right. A caller who sat in silence, got talked over, or could not catch a number has already had a failed call, and the transcript can still look correct.
On AVOX, the failures were conversational and integration failures, not reasoning failures. Turn detection fires early and the agent interrupts a caller mid-thought, or fires late and the caller repeats themselves. Background noise pushes the detector's confidence down on the calls where the customer is already frustrated. A switch between English and Hindi mid-clause produces a transcript fragment that is grammatical in neither language.
A text judge sees only the last of those failures, and sees it as a wording problem. The others leave no trace in the words at all, so a transcript-only eval files every one of them as a pass.
In the claim-status call, the text judge received one question and one answer, and the timestamps never reached it. The timeline shows a gap after the caller stopped, long enough that they began again, and then both sides speaking at once. If you add a line to the rubric asking whether the call "felt natural", the judge still has no audio, so it will infer a feeling from the prose. The complaints name waiting, being cut off, and not catching a reference number. The eval names answer quality, and the two lists never meet.
Check which half of the call any eval tool reads before you buy it, because replaying audio is not the same as scoring it. Microsoft's harness for its Voice Live API replays recorded calls through the agent, and its own documentation, updated in September 2026, has it score from the transcribed text with no word error rate or speech quality.
When is a call content-ok and still a bad experience?
A call is content-ok and experience-bad when the words answer the request and the caller still could not use the conversation. Content covers the facts, the policy and the tool result. Experience covers whose turn it was, how long the silence lasted, and whether the speech could be followed. Split the two, or a wording fix will take credit for a call the caller could not finish.
The figure names four outcomes of one call. It gives no frequencies, because the mix depends on your line.
| Cell | What the caller got | What a transcript judge says | What has to change |
|---|---|---|---|
| Content right, experience usable | A correct answer they could follow | Pass | The only cell a release may grow |
| Content right, experience unusable | The right words, on a turn they could not use | Pass, wrongly | An audio check has to fail this call |
| Content wrong, experience usable | A clear call with a wrong fact or a policy miss | Fail | A text rubric, one question in each judge |
| Both wrong | A wrong answer they also could not follow | Fail | Two defects, fixed on their own lines |
The second row misleads teams. Every transcript dashboard is green there, and the usual response is to leave the audio path alone and tighten the prompt. The caller could not take a turn, or could not hear the number they were asked to confirm, and a more careful sentence does not give them the floor back. Ship that prompt change and the cell does not move.
The fourth row has the opposite trap. Two checks fail, someone folds them into one number, and a large content miss drowns a smaller overlap. You ship a factual fix, the blended number jumps, and the overlap is still in the recording. A call leaves the second row only when the audio verdict changes, and it leaves the third only when the text verdict changes, so count them as separate exits.
Our guide to LLM evaluation covers finding failure modes by reading traces before you freeze a rubric. Do that reading on the recording, not only on the transcript. A failure category you never noticed while reading will never get a check written for it.
Which audio signals should you score?
Score the audio on four signals: when the turn was judged over, how an interruption was handled, whether names and numbers were intelligible, and where silence and overlap sat inside the turn. Each signal points at a different stage. One voice-quality number mixes them, so a change that shortens silence can hide a worse interruption, and you will not know which stage to open.
| Signal | Computed from | A failure means | It does not mean |
|---|---|---|---|
| End of turn | The detector's decision, set against the caller's next words | The agent started too early, or held on through a pause that was not an ending | That the answer text was wrong |
| Barge-in | Who spoke, whether playback stopped, and whether history was trimmed to what was heard | The agent talked through a real interruption, or it gave way to noise | That a tool call already sent was undone |
| Intelligibility | Word error against a reference, split out for names, numbers, dates and addresses | The words that drive the next action were not the words spoken or heard | That the whole conversation was the right one |
| Silence and overlap | Timestamps of each side, kept as a distribution per turn | A tail of dead air after the caller stopped, or overlap that was not a short acknowledgement | That the mean latency of the day was fine |
End of turn is the decision that the caller has finished. A fixed silence gap works on quiet, fluent speech and fails when the caller pauses to read out a policy number, or switches language mid-clause. Score the decision against what the caller did next. If they were still inside the same request, the detector cut in. If they had finished and waited long enough to start over, it held too long.
Turn stability is the turn-taking signal we have measured. On AVOX Realty, in conditions ranging from quiet rooms to call-centre noise, it improved by 15 to 35 points against our baseline in internal testing.
No transcript score could have produced that number, because it is a property of the turns rather than the words. Our guide on how to build a voice AI agent covers how to build the detector. The eval needs only the detector's decision in the trace, so a later build can be compared on the same calls.
Barge-in is the caller speaking while the agent is still playing audio. Stopping playback is not the whole event. The history has to be trimmed to what the caller actually heard, and the interruption has to be classed as a backchannel or a new turn. Without that class you cannot tell a polite "mm" apart from a caller trying to take the floor.
A hum that cuts the agent off is a false yield, and a real question that the agent talks through is a missed yield. Score those as different failures, because the fix for one causes the other.
An interruption also does not roll back a tool call that has already been sent. A barge-in check that looks only at audio will pass a call where the refund already ran. Pair it with the tool log whenever the action is hard to undo.
Word error rate is the usual intelligibility measure, and it is narrower than teams treat it. Use one reference for the caller's audio, which checks transcription. For the agent's speech, re-transcribe what was played and compare it with the text the agent meant to say, which checks whether the voice could be followed.
A single rate on the whole call hides a wrong surname inside a cloud of small function words. On AnyTeam, word error rate fell from roughly 18% to 10% over nine rounds of tuning, in our own measurement. The largest gains came on filler words and industry terms, which a whole-call average would have blurred. Score names, numbers, dates, addresses and your own domain terms as their own slice, because those words select the wrong tool call.
Audit a sample of the marked errors before you trust a movement in the rate. Some of them will be the label, not the model, and a regression that is really a relabelling will send you into the wrong component.
Silence and overlap come from timestamps you can log on every turn, so they can run on all traffic rather than a sample. Keep the distribution, because a mean will look calm while a tail of turns leaves the caller in dead air. We do not give a silence cutoff to copy, because a gap your callers tolerate on one line is a hang-up on another, and on our two phone agents property buyers were less patient than insurance callers.
How do you keep audio scores from double-counting text judges?
Run text judges and audio checks on the same call, and store the two verdicts on separate lines. Text owns whether the answer, the policy and the tool arguments were right. Audio owns whether the caller could take the turn and follow the speech. The gate fails when either line fails.
This figure is a framework. It shows which verdict belongs to which check, and which team owns an audio failure. It is not a vendor's metric list, and it states no weights.
Double-counting is easy to build by accident. A judge asked whether the call was helpful will punish a wrong refund amount, and a second judge asked whether the task completed will punish the same amount. Put a turn-taking score into that average and the factual miss dominates it. The blended number will rise when you edit the answer, and the turns that collided will still collide.
The evals guide asks for one question in each rubric, as the evals glossary entry explains. Voice adds a second rule: do not ask the text judge about silence, overlap or intelligibility when a timestamp or a word-error slice already records them. Asking an LLM as judge to rate the "naturalness" of a call whose silence you already measured puts the silence in the score twice, and you cannot tell which copy moved.
Keep the scale itself away from the model as well. On RepliCoach, a language model writes the qualitative feedback on each practice session, and a versioned rubric of weighted dimensions sets the number.
We separated them because if the model both writes the feedback and sets the number, a model upgrade shifts every historical comparison. Raw model scores there also drifted between sessions, conversation lengths and personas, so the rubric normalises them before they are compared.
Do not read a small move in a blended voice score as evidence, because judges vary on their own. Microsoft's harness documentation puts its run-to-run variance at 0.3 to 0.5 points on a 1 to 5 judge scale and tells users to average at least three runs, so a move smaller than that is noise.
The two lines also map onto two owners, as in our guide to AI agent monitoring. If the turn detector ran as configured and the call was still unusable, that is a quality failure owned by the team that builds the agent. If a stage did not run, audio never arrived, or synthesis returned nothing, that is an infrastructure failure owned by whoever is paged for the path. Our guide to analytics agents and text to SQL makes the same split between whether the answer was right and whether each check fired.
Jev, TypeSafe's decision model, can sit beside these lines as a check on whether a caller should reach a person, run off the request path and in shadow first. It reads text, not audio, so it cannot be the turn-taking eval, and its score must never override a caller who has already asked for a person.
How do you prove a voice fix on the same calls?
Prove a voice fix by replaying the candidate against a frozen set of recorded calls. The same audio, the same labels and the same traces go through every build you compare. The change holds when calls that failed an audio check now pass, calls that passed still pass, and the text verdicts are not worse. A synthetic call can suggest the change, but it cannot be the evidence that passes it.
This figure is a framework for the order of steps before a voice change takes live traffic, and it is not a pass rate from any line.
A frozen call is a file you can replay: the caller audio, the agent audio when you have it, the transcript, the end-of-turn and barge-in decisions, the tool trace, and the label for which cell the call was in. Version that set the way you version the prompt. If someone edits a label after the change, the comparison measures the edit.
Capture those events while the call happens. On RepliCoach, turns, speakers and timestamps are written as structured events at the time of speaking, because rebuilding them from audio afterwards was both lossy and too slow. Gaps in capture are silent. A dropped event produces a session that scores fine and is missing the exchange that mattered.
On AVOX, latency was timed per stage, so a slow call could be pinned to detection, transcription, reasoning, a tool or synthesis. On AVOX Realty the same timing showed that the extra time on Hindi and Hinglish calls was spent in transcription, not reasoning. A frozen set with per-stage timing turns "the new build feels slower" into a claim about one stage.
The candidate runs on those files, and you compare the verdicts call by call rather than the average. An average can rise because a group of easy calls got slightly cleaner wording while the collision calls stayed broken. Read the calls that flipped and count the direction on the calls you already labelled, which is the sign test from our evals guide.
| Line | Passes when | Fails when |
|---|---|---|
| Audio failures | Each call labelled experience-bad on a named signal now passes that signal | Any labelled failure is still failing |
| Regressions | Calls that passed the audio checks still pass them | A passing call grows a new silence, overlap or cut-in |
| Text | Task, policy and tool verdicts are unchanged or better | An edit made an answer or a tool argument worse |
Synthetic callers are useful before you have the recording, because they can force an interruption or a noisy room you want to see once. They speak the way they were prompted, and your callers do not. A simulation that improves is a reason to try the change on the frozen set, not a reason to widen traffic.
When live traffic shows a failure the set lacks, add that recording and freeze its label before the next comparison. Adding it after you have seen the new build's output is how a set starts agreeing with the change you hope to ship.
Release then widens by share of live calls, as in our architecture guide. The frozen set cannot contain a failure you have never recorded, and the share limit is how you find one without exposing every caller to it.
What does the voice architecture change about the eval?
The questions you ask of a call stay the same across architectures, and the fields you can score do not. A cascade gives you text at each stage. A speech-to-speech model gives you audio and a side transcript. A managed pipeline gives you whatever the vendor exports. Score the signals you can actually log, and do not treat a missing field as a pass.
A cascade runs speech to text, a language model and text to speech as separate stages, so you can fail transcription, the answer and the spoken audio on their own. Word error on the caller audio checks transcription, the text judge reads the model output, and a check on the played audio tells you whether the voice spoke what the model wrote. When a call fails, the stage is named.
A speech-to-speech model hears and answers in audio, and the transcript is a side output. In OpenAI's Realtime API it comes from a separate transcription model, and OpenAI's cookbook warns that the two models can disagree. A text judge on that transcript is checking a copy. Lean on the audio signals, and mark any content verdict that was made on a side transcript.
A managed pipeline runs the audio path for you and returns what it chooses to expose. You can still score tool arguments and policy, because those stay on your side of the boundary. Turn decisions, overlap and per-stage time exist only if the vendor exports them, so export the recording and the timestamps into your own store at the end of each call.
| You need to score | Cascade | Speech to speech | Managed pipeline |
|---|---|---|---|
| Answer and policy | Model text | Side transcript, labelled as a copy | Whatever text the vendor returns |
| Caller transcription | Your speech-to-text output | Only if a caller transcript is exposed | Only if it is exported |
| Agent intelligibility | Played audio against the text you sent | Played audio, with no separate text stage | Played audio, if you can record it |
| Turn-taking and overlap | Your detector's decision and timestamps | Model behaviour, from the recording | Vendor events, if they exist |
An empty cell in that table is a gap in the gate, not a pass. If you cannot fill it, keep the class of call that depends on it with people until you can, or choose an architecture that logs it.
Common questions
How do you test a voice AI agent?#
Replay recorded calls from real callers through each build you want to compare. Score the transcript with text checks and score the audio separately for turn-taking, interruptions, intelligibility and silence, and pass a call only when both pass. Keep that frozen set fixed between builds. Synthetic callers can suggest a change, because they can force an interruption on demand, but they cannot be the evidence that passes it.
Why does a correct transcript still fail a voice agent?#
A transcript judge reads the words. The caller hears timing, interruptions, and whether a name or a number was clear. If the agent waited too long, talked over the caller, or spoke a number they could not catch, the words can be right and the call has still failed. Score those on the audio, and fail the call when either the words or the audio fail.
What is a content-ok, experience-bad call?#
The answer, read as text, is right and within policy, and the caller still could not use the call. Typical causes are a long silence after they stopped speaking, the agent talking over them, or speech they could not make out. A change that only improves the wording does not fix it, and a transcript score will keep passing it until an audio check is allowed to fail the call.
Is word error rate enough to evaluate a voice agent?#
No. Word error rate compares a transcript with a reference and counts the words that differ. It does not see turn-taking, silence, or whether the caller was interrupted. Use it to check transcription and, separately, the agent's own speech, and score names, numbers, dates and domain terms as their own slice, because one wrong identifier drives the wrong tool call.
How do you stop audio scores and text judges from double-counting?#
Give each check one question. Text judges own the answer, the policy and the tool arguments. Audio checks own the timeline and whether the speech could be followed. Keep the verdicts on separate lines and fail the gate when either line fails. A blended score lets a perfect answer hide a broken turn, and counts one wrong fact twice if two judges both punish it.
What goes into a frozen voice eval set?#
Recorded calls from your own line, with the audio, the transcript, the turn decisions, per-stage timing and the tool trace kept together, plus a label for which failure the call shows. Freeze the files and the labels, and replay every candidate through that same set. Add a call when production shows a new failure, and freeze its label before you score the build meant to fix it.
How should a voice change be released?#
Replay it on the frozen set first. It passes when the calls that failed the audio check now pass, the calls that already passed still pass, and the text verdicts are not worse. Then widen the share of live calls, and send a class of call back to people if that class starts failing. A synthetic call can suggest the change but cannot pass it.
Does a speech-to-speech agent need a different eval?#
The questions stay the same, and what you can log changes. A cascade gives you text at each stage, so you can score transcription, the answer and the spoken audio apart. A speech-to-speech model reasons over audio and its transcript is a side copy from a separate model, so a text judge is checking that copy. Score the audio either way, and mark any content verdict that came from a side transcript.
Where do turn-taking failures sit in production ops?#
If the turn detector ran as configured and the call still failed the caller, that is a quality failure, and the team that builds the agent owns the fix. If a stage did not run, the audio dropped, or synthesis returned nothing, that is an infrastructure failure, owned by the platform team. One combined voice-quality score sends the page to the wrong team.
Further reading
- How to build a voice AI agent. Cascade, speech-to-speech and managed pipelines, and the turn-taking design this guide does not repeat.
- AVOX, the insurance voice agent and AVOX Realty. The failure analysis, per-stage timing, and the turn-stability measurement.
- RepliCoach, the sales roleplay agent. A rubric kept separate from the model, and capture at the time of speaking.
- AnyTeam, on-device meeting transcription. Word error rate over nine rounds of tuning, and where the gains came from.
- AI agent monitoring. Quality and infrastructure as separate owners.
- LLM evaluation. Rubrics, judge checks, the frozen set and the sign test on flipped cases.
- Voice agents, evals, LLM as judge and speech to text.
- Microsoft, How to evaluate Voice Live agents, updated September 2026. Replay of pre-recorded audio, text-only evaluators, and judge variance between runs.
- OpenAI cookbook, Out-of-band transcription. Why a speech-to-speech model's side transcript can differ from what it heard.