Where AI Is Not Replacing Jobs

Voice AI’s hardest engineering problem is not recognising speech. It is knowing when speech has stopped.

OpenAI’s Whisper, the most widely deployed speech-to-text model in production, has a peculiar habit. Fed audio that contains no speech at all, it sometimes invents a closing phrase: “Thank you for listening.” The model is hallucinating a podcast. The bug is well-known to engineers who run the model in real conditions, and well-papered over by abstraction layers built specifically to hide it. The fix is mechanically simple: do not give Whisper silence to transcribe. That sounds easy until one tries to do it.

Listen for nothing

Voice activity detection, or VAD, is the unglamorous gatekeeper of every voice-AI system. It decides, in real time, whether a sample of audio contains a human voice. Get it wrong by being too generous and the downstream model wastes compute on coughs, fans and machinery — or worse, hallucinates speech that was never spoken. Get it wrong by being too strict and the model misses the start of a sentence, leaves the operator hanging and breaks the conversation. The acceptable error rate is, at production scale, vanishingly small.

The two open-source workhorses for this job are Google’s WebRTC VAD, released over a decade ago, and Silero VAD, a deep-learning model first published in 2021 and trained on more than 6,000 languages. Both are free. Neither is good enough. At a 5% false-positive rate — a level production teams generally tolerate — Silero correctly identifies 87.7% of speech frames, while WebRTC catches only half. In other words, the older library misses one frame out of every two it should have flagged. Silero, the better of the two, still misses 14% of real speech under conditions that engineers describe as clean. “Clean” here means a quiet room, a single speaker, a near-field microphone. A factory floor is none of those things.

The 90-decibel question

America’s Occupational Safety and Health Administration sets the permissible noise-exposure limit at 90 decibels for an eight-hour shift. The hearing-conservation threshold begins at 85. Mining, agriculture and parts of heavy manufacturing routinely operate at 93 decibels and above, with peaks well into the 100s. Manufacturing alone accounts for roughly 88% of OSHA’s logged noise-exposure measurements. Voice AI built for these environments is not building for an edge case. It is building for the median customer.

But the noise is the easy part. Modern beamforming microphones, spectral subtraction and neural noise-cancellation can clean an audio signal impressively. The harder problem is what happens after the noise is removed. Aggressive denoising eats the bottom of the speech spectrum along with the machinery. The very pipeline that strips out a 92-decibel air-handler also removes the soft consonants — t, k, p, f — that mark the end of a sentence. The model can hear the words. It cannot hear the silence between them.

The endpointing trap

This is the endpointing problem, and it is where most voice-AI startups quietly bleed. A correct endpoint — the system’s judgment that the speaker has finished — is what triggers the model to respond. Trigger too early and the agent interrupts the operator mid-thought. Trigger too late and the latency budget collapses, the conversation feels stilted and adoption craters.

Industry consensus puts the natural pause-tolerance for human conversation at roughly 200 to 500 milliseconds. The system has that long to decide whether the speaker has stopped, or merely paused to think. Most production teams approach this by stacking heuristics on top of VAD: an energy gate, a tail timer and a context-aware adjustment for who is speaking. The result works in benchmarks and breaks in plants. A 22-year fitter who pauses for four seconds mid-instruction to look at a gauge is not finished speaking. He is thinking. Cut him off, and the next time he sees the device, he ignores it.

Move fast and miss things

The conventional response is reasonable. Throw a larger model at the problem. Use a multilingual transformer trained on industrial audio. Add a second-stage classifier for environmental sound. The largest voice models on the market — those from Deepgram, Cartesia, ElevenLabs and OpenAI — already incorporate sophisticated noise-robust front-ends. The benchmarks improve every quarter.

The benchmarks lie about the failure mode. Production noise is not stationary. A drill press starts. A forklift reverses. A PA announcement triggers. Each event shifts the audio’s statistical baseline, and the noise-robust models trained on YouTube audio have never heard a Mazak 5-axis CNC at full draw. The transcription accuracy may hold. The endpointing collapses, because the model now sees noise where it expects silence — and silence where it expects noise.

Curtain call

The implication for builders — for voice-AI startups, ASR vendors and the industrial-software teams trying to deploy them — is unromantic. Voice AI for industrial settings will not be won by a better transcription model. It will be won by a smarter silence model: one that can distinguish a thinking pause from a finished thought, an active machine from a pending instruction and a worker who has walked away from one who is mid-handover.

That is a harder, less fundable problem. It will not produce a viral demo. It will produce a system that works.

Bell Labs spent decades on speech timing when voice recognition was a research curiosity. The current generation of startups is mostly skipping the problem and hoping the larger model will solve it. It probably will. But not, perhaps, in time. The next breakthrough in voice AI will come from someone who solves silence.