A voice model that starts speaking in under 50 milliseconds, and the serving code is open
Nari Labs published a write-up on 19 August of a serving stack it built for Qwen3-TTS 1.7B CustomVoice, and open-sourced both the implementation and the benchmark. On a single H100 SXM it holds a 95th-percentile time-to-first-audio under 50 milliseconds at 10 requests per second, and stays below 100 milliseconds even at 20. At full load the cost works out at roughly $2 per million characters of speech. The gains come from scheduling rather than from a new model: the three components of the system, the Talker, the Code Predictor and the codec, are scheduled as one unit instead of three separate queues, requests are prioritised by deadline so streaming work does not stall behind batch work, and decoding is incremental with cached state.
Why it mattersLatency is what decides whether a spoken interface feels like a conversation or a phone menu. Under roughly 200 milliseconds a person reads the reply as an answer; past half a second they start talking over it. Getting under 50 has generally meant a proprietary stack from a voice API company, priced accordingly. This is the trick that has been reshaping text inference for two years, that most of the speed was sitting in the serving layer rather than the weights, arriving in audio, in public, with the code attached. If you are building anything that talks, the floor for acceptable just moved.
Read in the app — free, in 9 languages
Related stories
Anthropic will let its strongest model hunt flaws in your code — but it will not let you talk to it2026-08-22Apple Music will tell you when a song was made by a machine — but the uploader decides whether to say so
2026-08-21Stripe has just paid $7.5 billion for a model router. Days later Ramp built one and is giving it away until January.
2026-08-21Meta's assistant is now a Mac app that reads your screen and types into any window — and what it sees can train the model
2026-08-21One click on a news site now tells Google to show you more of it — and you stay on the page you were reading
2026-08-21