Voice AI: when is the "Her" moment? — Neil Zeghidour, Gradium AI
Neil Zeghidour, CEO of Gradium AI, deconstructs the gap between current voice AI and the "Her" ideal. He argues that while cascaded systems are practical, they are architecturally flawed for natural conversation. The future lies in full-duplex, speech-to-speech models that not only solve latency but also integrate deep paralinguistic understanding and overcome significant cost barriers.
The “Her” moment has become an overused analogy in voice AI, yet it remains a relevant benchmark for what a truly natural human-computer conversation should feel like. Despite numerous demos, we are still far from this ideal. The current state of voice AI is defined by a core tension between two architectural approaches: cascaded systems and end-to-end speech-to-speech models.
The Architectural Divide: Cascaded vs. Speech-to-Speech
The most common architecture today is the cascaded system, which chains together three components:
- Speech-to-Text (STT)
- Large Language Model (LLM) for intelligence and reasoning.
- Text-to-Speech (TTS) for generating the audio response.
These systems are practical, convenient, and increasingly intelligent. However, they are architecturally incapable of achieving the low latency required for a natural conversation. A typical human conversational turn takes about 200 milliseconds for understanding, processing, and responding. In contrast, even a fast TTS model alone can exceed this 200ms budget.
This latency problem is massively compounded by tool calls. When an agent needs to access an external API or database, the latency can unpredictably jump to anywhere between 500ms and 4 seconds. This makes the interaction feel slow and disjointed. One workaround is to use “fillers”—having the LLM generate conversational text to bridge the gap while waiting for the tool to return a result. This can make the wait feel more natural, but it doesn’t solve the underlying architectural limitation.
“We are fighting for latency of the TTS, trying to grab 10 milliseconds, 20 milliseconds, and then you have a tool call… that is going to have a latency between 500 milliseconds and 4 seconds. So in a way, sometimes you feel like we’re fighting for something that is not the most relevant anymore.”
The Duplex Dilemma: Why Conversations Still Feel Robotic
The alternative is the speech-to-speech (S2S) model, which processes audio input directly to audio output, bypassing the intermediate text steps and significantly reducing latency. However, nearly all current S2S models suffer from a critical flaw: they are half-duplex.
- Half-duplex means the model is either listening or speaking; it cannot do both simultaneously. This prevents natural conversational behaviors like interruption and “backchanneling” (the “mhm,” “uh-huh,” and “yeah” utterances we use to show we’re listening).
- Full-duplex allows for overlapping speech, which is essential for human-like interaction. In some cultures, conversational overlap can occur up to 20% of the time.
The lack of full-duplex capability is why even the most advanced voice models can feel frustratingly robotic, often stopping abruptly when interrupted or talking over the user without acknowledging them.
The Moshi model, developed by Gradium’s non-profit lab, was the first demonstration of a truly full-duplex conversational system. It could handle interruptions and overlapping speech gracefully, creating a conversational flow that remains unmatched.
Beyond Flow: The Unsolved Challenges
While Moshi solved the duplex problem, it revealed a new set of challenges. The model achieved a natural flow but was ultimately “stupid” and “useless.” It had no real intelligence, no ability to perform tasks via tool calls, and no observability, making it impossible to use in a production environment.
This highlights the central challenge: we must imbue these fluid, full-duplex models with the same reliability, intelligence, and personalization as today’s cascaded systems.
Furthermore, a critical piece is missing: paralinguistic understanding. Voice carries a rich layer of information—tone, hesitation, emotion, sarcasm—that is completely lost when speech is converted to text. A true “Her”-like AI must not only produce natural-sounding speech but also understand the non-textual cues within the user’s voice. This is not a prompt engineering problem; it is a fundamental science problem that requires models trained to leverage this paralinguistic data.
The Practical Barriers: Cost and Privacy
Hiding behind the latency problem is another massive wall: cost. Voice processing, particularly high-quality TTS, is extremely expensive. The voice modes of most hyperscalers currently run at a loss. For startups building consumer voice applications, TTS bills can consume their entire fundraising before they can achieve a significant user base.
To make voice AI scalable and profitable, the cost must be driven down. This points toward on-device processing. Running models locally on a device’s CPU, like a smartphone, offers two key benefits:
- Cost: It eliminates API fees, allowing for massive scale without corresponding costs.
- Privacy: It keeps sensitive user conversations on their device, which is becoming increasingly important to users.
Gradion’s Phonon is a step in this direction—an on-device TTS model that runs entirely on a smartphone CPU, providing high-fidelity speech with voice cloning capabilities without cloud dependency.
The Path Forward
Voice is not a solved problem or a commodity. The final steps toward truly natural conversational AI will be the most difficult. The path forward requires deep scientific and engineering work to create models that are:
- Full-duplex for natural, interruptible conversation.
- Intelligent and useful, with robust tool-calling capabilities.
- Perceptive, with a deep understanding of paralinguistic cues.
- Scalable and affordable, primarily through efficient on-device processing.