Every hard problem in this system comes from one decision: the audio does not leave the machine. That removes the buyer's objection and takes away every convenience a cloud pipeline has, unlimited compute, a clean single audio source, and the freedom to be slower than the conversation.
This is the engineering half of the case study. The business read (what changed for the reps using it, and why the privacy boundary was a commercial requirement) is a separate document.
The problem
Four constraints, and they fight each other.
The compute budget is whatever a video call has not already taken. Not "a laptop". A laptop that is currently encoding and decoding video, running a conferencing app, and doing whatever else the rep has open. Everything below has to fit in what remains, on CPU, without the fans making the rep the loudest person in the meeting.
The input is not one clean stream. It is system audio and microphone audio, from an unpredictable combination of conferencing apps, headsets, laptop speakers and external interfaces, with loopback to handle and no guarantee the two sides arrive aligned.
The transcript has a downstream contract. It is not being read by a human who can mentally filter disfluency. It is being written into a CRM and injected into prompts. Filler words, repeated stutters and hallucinated phrases do not degrade that output gracefully; they break it.
And detection had no reference implementation. Reliably knowing a meeting is live is the capability the entire product depends on (if it misses, nothing else in the pipeline ever runs) and at the time only a handful of applications anywhere had solved it. None of them had written down how. We designed and tested that path from first principles inside the desktop app, against OS events and audio state rather than against a spec.
-
Under real-time on CPU, alongside an active video call
-
Two audio sources, synchronised, across hardware we do not control
-
Transcript clean enough to automate against, not just clean enough to read
-
Detection that recovers from partial states, because a missed start means the product did nothing at all
Architecture
Four layers, each isolated so a failure in one is diagnosable rather than presenting as "the AI is wrong".
A context layer resolves the meeting before it starts: calendar invite in, companies and participants resolved, structured brief out. This is the cheapest possible quality win, the model reasons over a known account rather than inferring one from a transcript.
A native capture layer takes system and microphone audio, handles loopback, separates the two, and feeds a single synchronised stream forward. This is native code rather than anything in the app runtime, because it is the layer where consumer hardware variability actually lands.
An on-device transcription layer runs a compact, quantized speech model, CPU-only, embedded in the app. Decoding is biased toward sales and product vocabulary, which is where the accuracy that matters commercially actually lives: getting a product name or a competitor right matters more to a deal record than getting a preposition right.
A streaming layer carries cleaned transcript segments and the context brief into language-model flows for live guidance, then produces the structured post-call record (entities, risks, next steps) in machine-readable form.
The business read: why the privacy boundary was the product decisionDesign decision: transcript cleaning sits before the model, not after it. Everything downstream (guidance, summary, CRM fields) inherits the transcript's noise, so cleaning once at the source is worth more than repairing three times at the consumers.
How it works
Resolve the meeting#
The calendar invite is read ahead of the call and enriched into a structured brief of companies and participants, so context exists before the first word is spoken rather than being inferred from the transcript afterwards.
Detect the live call#
Detection combines OS-level event handling with audio-state checks, and (the part that took the work) recovery from partial states. A meeting that starts early, a call joined from a second device, an app that reports a state it is not actually in: each of those has to resolve to the right answer, because a missed start means the product produced nothing at all for that call.
Capture and separate#
The native module captures both system and microphone audio, handles the loopback path, separates the sources cleanly, and emits one synchronised stream. Real consumer hardware is the adversary here, not the algorithm.
Transcribe on CPU#
The embedded quantized model transcribes locally, with decoding biased toward the vocabulary the business actually depends on. Nine rounds of optimisation is what moved this from "works on a quiet machine" to comfortably under real-time on commodity laptops during an active call.
Clean, then stream#
Cleaned segments stream with the context brief into the model flows that return live prompts, and after the call into the extraction that produces the structured CRM record.
Ship it as software#
Signed and notarized desktop app, one codebase shared across web and desktop, distributed outside the app stores. Real distribution constraints, solved rather than deferred, a local AI pipeline that cannot be installed by a normal sales team is a research project.
Control planes
Governance#
The device is the boundary, enforced by architecture rather than policy: there is no path by which captured audio reaches a cloud service. Meeting identity is established from calendar, company and participant context rather than inferred. Structured CRM fields are a defined contract, kept separate from raw audio and free-form transcript, so what lands in a business system is a record rather than a dump.
Orchestration#
Context resolution, detection, dual capture, stream synchronisation, local transcription, guidance streaming, post-call extraction and CRM handoff all run concurrently, inside the compute an active video call leaves over. The orchestration problem here is a scheduling problem, not a prompting one.
Observability#
The pipeline reports detection state, audio routes, capture interruptions, transcription timing, model resource use, guidance latency, extraction results and CRM handoff.
This is what makes the system supportable. Every user-visible symptom on a product like this presents identically ("it didn't work") and without per-stage state, every one of those tickets becomes a model investigation when most of them are an audio-routing problem.
When it fails
The failures are hardware and state failures, not model failures.
A meeting starts and detection misses the transition. A rep swaps a headset mid-call and the capture path changes underneath the stream. The conferencing app reports a state that has already moved on. Two sources drift out of alignment. In each case the pipeline has to re-establish state rather than continue producing confidently wrong output, because a transcript from a half-connected audio path is worse than no transcript. It will still be written to the CRM.
That is why detection is specified as recovery from partial states rather than as a boolean. And it is why per-stage state reporting is not an operational nicety here: a capture interruption and a transcription slowdown produce the same complaint from the rep, and only the instrumentation tells them apart.
Results
The pipeline runs where the call happens, keeps pace with it, and produces output clean enough to write to a business system unedited.
-
Comfortably under real-time on commodity laptops, across nine rounds of optimisation, alongside an active video call
-
Word error rate from ~18% to 10%, concentrated on filler words and industry vocabulary, the failures that broke downstream automation
-
On-device live-meeting detection, shipped inside a signed and notarized desktop app
-
No upload path for customer audio, which makes the privacy claim architecturally verifiable rather than contractual
Evidence
Directly demonstrable in the shipped product: live-meeting detection, local transcription, the embedded model, the signed and notarized desktop application, and the absence of a cloud audio path.
Internal measurement not yet re-confirmed on current production traffic: the word-error-rate change from roughly 18% to 10%.
On the detection claim: that only a handful of applications had solved live-meeting detection, and that none had documented it, is our own assessment from the period when we built it. We stand behind it (it is why the work took what it took) and we would rather state it plainly than round it off. It rests on our survey of the field at the time rather than on a published ranking, and we will say so to anyone who asks.
If you are evaluating a local-first pipeline of your own, the question worth asking early is not which speech model to embed. It is what your transcript's downstream consumer will do with a disfluency, and whether you can tell an audio-routing failure from a model failure at three in the afternoon on a support call.
How we classify these numbers, and the value model behind them