Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral
Samuel Humeau from Mistral explains the dominant architecture for modern text-to-speech (TTS) systems, which mirrors large language models. He details how neural audio codecs solve the information density problem, the autoregressive transformer backbone for generation, and the streaming techniques used to achieve low perceived latency in voice agents. The talk uses Mistral's open-weight TTS model as a practical example.
The architecture of modern text-to-speech (TTS) systems is converging on a dominant pattern, heavily inspired by the success of large language models. The goal is to reframe speech generation as a sequence modeling task, where an autoregressive transformer generates audio one piece at a time. This approach is particularly crucial for the primary use case of TTS today: voice agents.
Voice Agents and the Primacy of Latency
For voice agents, which typically use a pipeline of Speech-to-Text (STT) -> LLM -> TTS, latency is the most critical factor for a good user experience. A key technique to reduce perceived latency is streaming. While the full audio might take seconds to generate, the system can start playing the first few audio packets as soon as they are created. This makes the agent feel much more responsive.
As Samuel Humeau demonstrates with an agent built using a Mistral LLM and their new TTS model, this allows for a fluid conversation even with a slight delay in the full audio generation.
Solving the Information Density Problem with Neural Audio Codecs
A fundamental challenge in audio generation is the sheer amount of data. Raw audio has a very high bitrate (e.g., a standard MP3 is ~200,000 bits per second), whereas the core semantic information is much lower (human speech is only ~15 bits per second). Feeding raw audio waveforms to a transformer is computationally infeasible.
The solution is to use a neural audio codec. This model acts as an encoder-decoder system with a bottleneck, trained to solve two problems:
- Compression: It transforms a frame of audio (e.g., 80 milliseconds) into a small set of discrete tokens. This reduces the data rate to a manageable level, such as a few thousand bits per second.
- Information Retention: Through its training process, which involves reconstruction and adversarial losses, the codec learns to discard irrelevant noise and retain the essential acoustic features that define a voice, tone, and prosody.
In Mistral’s model, each 80ms frame is converted into 37 tokens, resulting in a sequence of about 500 tokens per second for the main model to process.
The LLM-like Architecture of Modern TTS
The complete system generally follows a three-part structure: an audio codec, a core backbone model, and a decoder.
- Backbone: The core of the system is a large autoregressive transformer (4 billion parameters in Mistral’s case). Instead of generating one audio sample at a time, it generates one frame of audio tokens at a time.
- Frame Generation: A common pattern is to have a smaller, secondary transformer that generates the multiple tokens within a single frame. However, Mistral’s released model takes a different approach, using a diffusion model (specifically, a flow matching model) to generate all 37 tokens for a given frame at once.
- Conditioning: The backbone is conditioned on the input text and, for voice cloning, a few seconds of a reference audio clip. This context guides the model to produce the correct words in the desired voice. The model demonstrates impressive zero-shot voice cloning capabilities, even inferring a realistic accent when generating English speech from a French voice prompt.
Open Challenges: Streaming Text Input
While streaming output audio is a solved problem, the next major frontier for latency reduction is handling streaming input text. In an agent pipeline, the LLM generates text token-by-token. An ideal TTS system would start generating audio as soon as the first few tokens arrive, rather than waiting for the entire text response to be completed.
This is still an open research problem with two emerging patterns:
- Interleaved Streams: A single model processes a combined sequence of audio and text tokens.
- Dual-Stream Architecture: Separate streams for audio and text are processed and blended during inference.
Solving this challenge is key to creating truly seamless, low-latency conversational AI.