An adversarial roleplay loop with normalised scoring

An adversarial agent has to interrupt, and interruption is a real-time judgement. A cooperative agent waits for a turn to end. This one has to decide mid-utterance whether the rep has said something worth pushing on, and cut in, early enough to feel like a real objection, late enough not to be nonsense. There is no end-of-turn signal to wait for, because the whole behaviour happens before the turn ends.

Scoring needs structure that audio does not have. A recording cannot be scored consistently, compared across reps, or tracked over time. It has to become turns, speakers, timestamps and events at capture time, reconstructing that later from audio is both lossy and too slow to be part of a practice loop.

Rubric scores are not comparable by default. Raw model scores drift between sessions, between conversation lengths and between personas. A hard persona producing lower scores than an easy one is correct; a hard persona producing lower scores for the same performance is a broken measurement, and the two look identical without normalisation.

And the feedback loop has to close fast enough to be practice. Feedback the next day is feedback about a call the rep has stopped thinking about.

• Barge-in decided mid-utterance, not after a turn boundary

• Structured capture at the time of speaking, not transcription afterwards

• Normalised scores, comparable across personas, lengths and reps

• Multi-tenant concurrency, because a sales floor practises at the same time of day

Three pieces on a shared runtime, and the runtime is the same one behind our insurance and real-estate agents, which is why a third domain cost a tool layer and a rubric rather than a build.

The agentic voice runtime is a streaming roleplay loop with turn segmentation and role tagging. What is specific to this product is the objection policy: the logic deciding when to push, what to push on, and how hard, given the persona and deal stage. That policy operates against partial utterances, which is what makes it a different real-time problem from a cooperative agent.

The conversation capture layer emits structured events as the conversation happens (timestamped turns, speakers, session metadata) into a unified output schema. Capture is a first-class concern rather than a byproduct, because everything downstream is a function of it.

The scoring engine is schema-driven: weighted dimensions, behavioural markers and normalisation logic, applied identically to every session. Communication, persuasion, product knowledge and closing, plus strengths and missed opportunities. Language models produce the qualitative read (summary, gaps, objection detection) and the rubric produces the number, deliberately kept separate so a model change cannot silently move a score's meaning.

A multi-tenant data plane with normalised schemas and API access feeds dashboards and BI directly.

Design decision: the qualitative analysis and the score come from different mechanisms. If the model both writes the feedback and sets the number, a model upgrade shifts every historical comparison and nobody can tell whether reps improved or the scale moved.

Configure the pressure#

Persona, objection profile and deal stage set what the agent will push on, so practice targets a specific weakness rather than being generic.

Run the adversarial loop#

Streaming roleplay with turn segmentation and role tagging. The objection policy evaluates the in-flight utterance and decides whether to interrupt, which is the behaviour that makes the practice worth anything.

Capture as structured events#

Turns, speakers, timestamps and metadata are written as the conversation happens, into one schema, so the session is queryable the moment it ends.

Analyse qualitatively#

Models produce the session summary, strengths and gaps, objection detection and coaching feedback from the structured transcript.

Score against the rubric#

Weighted dimensions and behavioural markers, then normalisation, so the number is comparable to the rep's last session and to their colleague's.

Serve it#

Rep- and team-level metrics through APIs and dashboards, multi-tenant, with many roleplays running concurrently.

Governance#

The rubric is the authority on scoring, not the model. Weights, behavioural markers and normalisation are defined and versioned, so a score means one thing across reps and across months. Keeping the model out of the scoring path is the control that matters: it means a model change alters the quality of the written feedback without moving the scale underneath the historical data.

Orchestration#

Per turn: segment, tag the role, evaluate the objection policy, decide barge-in, respond. Per session: capture events, analyse, score, normalise, publish. Multi-tenant throughout, since a sales floor practises in the same hour.

Observability#

Every session resolves to its turns; every score resolves to the behavioural markers that produced it. That second path is what makes the system usable rather than merely correct, a rep who disagrees with a score can see what drove it, and a coaching tool that cannot show its reasoning gets argued with instead of acted on.

Score distributions per persona are the calibration signal. If a persona's distribution drifts, the persona has changed difficulty, and the normalisation needs revisiting before the scores are read as a trend.

Barge-in timing is the failure the user feels. Too eager and the agent talks over a rep mid-sentence for no reason, which teaches the wrong lesson and breaks the simulation. Too slow and it is not an objection any more, it is a comment. The policy has to be tuned per persona, because an aggressive buyer and a sceptical one interrupt at genuinely different moments.

Capture gaps are the failure that is silent. A dropped event produces a session that scores fine and is missing the exchange that mattered, which is why capture writes as the conversation happens rather than being assembled from audio afterwards.

And normalisation drift is the slow one. A persona whose difficulty shifts, or a rubric edited without versioning, makes historical comparison quietly meaningless while every individual score still looks reasonable.

Measured on our own R&D build.

• Manager review from 20–40 minutes to 2–3, with a guided per-session summary replacing full-call listening

• Scoring variance across evaluators sharply reduced, because the rubric replaces the reviewer as the authority

• Sessions queryable as structured data, not recordings, so trends across reps and time are computable

• The voice runtime reused across a third domain, needing new tools and a new rubric rather than a new build

Directly demonstrable: the real-time adversarial roleplay loop with turn segmentation and role tagging, structured session capture into a unified schema, the weighted rubric with normalisation, model-generated qualitative analysis, and the multi-tenant data plane behind dashboards and APIs.

Internal results, ours alone: the review-time reduction and the variance improvement. This is a Bigcircle R&D build with no client and no deployment, so nothing here is a customer-confirmed result.

If you are building evaluation on top of a model, the separation worth copying is between the thing that writes the feedback and the thing that sets the number. Merge them and your scores stop being comparable the first time you change models, which you will, and you will not find out until someone asks whether the team actually improved last quarter.