Speech recognition demos are staged with a single speaker, a good microphone, a quiet room and vocabulary the model has seen a million times. Real meetings have four people on a laptop microphone, one dialling in from a car, an internal product name that appears in no dictionary, and two of them talking at once for the most important thirty seconds of the call. Building MeetWise AI has meant working through the gap between those two situations, and the gap is where all the engineering lives.
Latency and accuracy are the same dial
A streaming recogniser must decide when to commit to a word. The more audio it waits for, the more context it has and the better its guess — future words genuinely disambiguate past ones. Commit early and the transcript appears responsive but wrong; commit late and it is accurate but arrives after the conversation has moved on.
Most production systems handle this by emitting provisional results that get revised. The user sees text appear almost immediately, then watches some of it change as more audio arrives. That is a reasonable engineering answer and a genuine user experience problem: revisions are distracting, and a transcript that rewrites itself undermines confidence in the whole feature. The design choices that help are visual rather than acoustic — render unstable text in a lighter weight, revise only within the current utterance, and never revise a line the user has already scrolled past or acted on.
The more important realisation is that live transcription and the final transcript are different products with different requirements. The live view exists so people can follow along, catch up after distraction, and see that the system is working. The stored transcript is what summaries, action items and search are built on. There is no rule saying they must come from the same pass. Running a second, non-streaming pass over the recorded audio afterwards — with full context, no latency budget and a larger model — produces a materially better artefact for everything downstream. Users experience it as a transcript that “tidies itself up” after the call, which is a feature rather than an apology.
Diarisation: who said that
Separating speakers is a harder problem than recognising words, and it is the one that most often makes a transcript useless. A summary that attributes a commitment to the wrong person is worse than no summary, because it is confidently wrong and someone will act on it.
Several things make diarisation difficult in ordinary meetings. Speakers with similar voices get merged. A speaker who changes tone — presenting, then joking, then reading a number off a screen — gets split into two identities. People move relative to the microphone. Someone joins halfway through, which means the number of speakers is not known in advance, and algorithms that need that number up front degrade badly when you guess wrong.
Streaming makes it harder still. Offline diarisation can cluster voices across the whole recording; a live system must assign a label before it has heard the rest of the meeting, so early labels are the least reliable ones.
The pragmatic answers are worth stating plainly. Where the platform gives per-participant audio channels — most conferencing systems do — use them, because channel-based attribution is far more reliable than clustering and the acoustic problem largely disappears. Where it does not, treat labels as provisional and reconcile them in the post-call pass. Let users rename and merge speakers, and make that correction persist for future meetings. And distinguish, in the interface, between “we know who said this” and “we think it was this person”: a system that hides its uncertainty spends the user’s trust on your behalf.
Vocabulary the model has never heard
Every organisation speaks its own dialect: product code names, internal acronyms, client names, supplier names, a system everyone calls by a nickname. General recognisers map these to the nearest plausible common word, which is exactly the wrong behaviour, because those are the highest-information words in the transcript. Losing a filler word costs nothing. Turning a client’s name into something else corrupts the summary, the search index and the action items in one step.
The available levers, roughly in order of effort:
- Custom vocabulary or biasing. Most recognition APIs accept a term list that raises the probability of those words. Cheap, effective, and it degrades if the list grows unfocused — bias everything and you bias nothing.
- Context-specific lists per meeting. The attendee names, the calendar title, the account being discussed. This is the highest-value version because the list is short and genuinely relevant to the audio in question.
- Learning from user edits. When someone corrects a term in a transcript, that is a labelled example. Feeding those corrections back into the biasing list is how the system gets better at a specific organisation over time.
Accents, noise and the honest limits
Recognition quality varies by accent. This is a property of how models are trained rather than a bug to be fixed with a configuration flag, and any vendor claiming otherwise is selling. For a product used across the UK and internationally, it means accuracy is not one number — it is a distribution, and some of your users sit in the tail. Testing on audio that resembles the audio your customers actually produce, including regional accents and non-native speakers, is the only way to know where you stand.
Noise and hardware are the other half. Laptop microphones with aggressive noise suppression sometimes remove speech along with the noise. A speakerphone in a hard-surfaced room adds reverberation that smears word boundaries. Someone on a mobile connection loses packets, and packet loss produces a specific transcript failure: text that stays fluent and grammatical while the speaker’s actual words are missing. That is worse than an obvious gap, because nothing downstream can detect it.
This is why confidence scores should be carried through the pipeline rather than discarded once the transcript is rendered. An action item extracted from a passage the recogniser was unsure about should not be presented with the same certainty as one from clean audio.
Interruptions and overlapping speech
People interrupt. They finish each other’s sentences, agree audibly while someone else is talking, and the moment of genuine disagreement in a meeting is very often the moment two people speak at once. Most recognisers handle a single stream of speech; overlapping speakers produce either a merged, garbled segment or the loss of the quieter voice entirely.
Per-participant channels solve much of this too: the decision to capture separate streams pays back across several otherwise-hard problems. Where you only have a mixed room recording, a transcript that marks a passage as crosstalk is more useful than one that invents a clean sentence.
Privacy is a design constraint, not a policy page
Meeting audio is among the most sensitive data a business produces. It contains salary discussions, commercial terms, personnel matters, health information disclosed in passing, and the opinions people hold about each other. Under UK GDPR it is personal data, and voice recordings raise additional questions because a voice identifies its speaker.
The decisions that matter are structural and made early:
- Where audio is processed, and by whom. Using a third-party recognition API means audio leaves your infrastructure. Customers will ask which provider, in which region, under what terms, and whether their audio can be used for training. Have the answers documented before the question arrives.
- Whether raw audio is retained at all. Keeping recordings enables reprocessing with better models; it also creates a liability that grows every day. Deleting audio once the transcript is produced is a defensible default, and a retention period the customer controls is better still.
- Consent and notification. Everyone in a meeting is a data subject, including participants who are not your customers. Visible, unambiguous indication that recording and transcription are active is the minimum, and in several jurisdictions rather more is required.
- Deletion that reaches everything. A deletion request must remove the transcript, the summary, the search index entry, the embeddings and the backups. If any derived artefact survives, the deletion did not happen.
What to ask a transcription vendor
If you are evaluating rather than building: ask for accuracy figures on audio like yours rather than on a benchmark, and supply the sample yourself. Ask whether diarisation uses separate channels or acoustic clustering. Ask how custom vocabulary is supplied and how many terms it holds. Ask what happens to your audio after processing, in writing. Ask whether confidence data is exposed through the API, because if it is not, you cannot tell your users when to double-check.
Then test with your worst recording, not your best one. The best one always works.