Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI
Bohan Li of EliseAI unveils a sophisticated architecture for real-time voice agents that merges high intelligence with low latency, akin to a self-driving car's perception, planning, and control systems. Key innovations include a speculative transcriber combining fast streaming with accurate contextual correction, background agents for offloading LLM tool calls and enabling eager generation, and a clever prefix cache for speech synthesis that seamlessly blends pre-cached audio with dynamically generated speech to mask underlying processing delays.
Bo Li from EliseAI details the architecture of their real-time voice agent harness, designed to achieve “Frontier Intelligence” without sacrificing speed, even when relying on inherently slow Large Language Models (LLMs). He draws a powerful parallel to self-driving cars, mapping the voice agent stack to:
- Perception: Transcription, converting real-world signals into data for the language model.
- Planning: The language model processing data and generating responses.
- Control: Speech synthesis, turning text into audio for the agent’s output.
The core approach involves applying specific optimization tricks at each of these cascaded layers.
Perception: The Streaming Speculative Transcriber
To achieve both speed and accuracy in transcription, EliseAI employs a “streaming speculative transcriber” system, layering two distinct transcription engines:
- Fast Streaming Transcriber (e.g., Flux): This engine provides immediate, initial detections, ensuring real-time responsiveness.
- Slower, Accurate Batch Transcriber (e.g., Scribe V2): This engine processes more context, offering higher accuracy.
The system intelligently combines their outputs:
- Initial text from the fast transcriber is used immediately.
- If the slower engine confirms the fast transcriber’s output, no correction is needed.
- Crucially, if new audio arrives, it effectively “cancels” any pending, older corrections from the slower, more accurate engine, prioritizing the freshest information.
- The Scribe V2 layer, leveraging the context of the user’s current query (e.g., “Can you provide your name and date of birth?”), is capable of understanding semantic types, allowing it to accurately extract entities like names and dates of birth even from imperfect initial transcriptions.
Planning: Eager Generation and Background Tool Calls
Recognizing that slow but intelligent LLMs and frequent tool calls introduce significant latency, EliseAI implements two key strategies:
- Background Tool Calling: Instead of the main LLM making synchronous tool calls, a background agent quietly executes these calls. The results are then injected into the main model’s context, making the LLM “think it made the tool call” itself. This “sleight of hand” drastically reduces round trips and latency for the primary model.
- Eager Agent Generation: The agent generates early responses based on partial, streaming transcriptions. These eager generations are held back and not immediately emitted. If a background tool call (e.g., successfully extracting a name and date of birth) provides critical new context that invalidates the eager generation, that generation is canceled, and the agent re-triggers with the enriched context. This allows for dynamic correction and integration of crucial information (like correcting mis-transcriptions or performing phonetic matching for names) without observable delay to the user.
Control: Streaming Speech Synthesis with a Prefix Cache
The final layer, text-to-speech, aims to hide the latency of full text generation by playing audio even before the entire response is ready. The “neatest trick” here is the prefix cache:
- Prefix Cache Mechanism: This cache monitors the agent’s output stream for sequences of words (prefixes) for which audio has already been generated in previous turns or earlier in the current turn.
- Instant Playback: When a cache hit occurs (e.g., for common opening phrases like “You said your name is…”), the cached audio is played immediately.
- Concurrent Synthesis: While the cached audio plays, the rest of the sentence continues to be generated by the LLM and streamed to a standard text-to-speech provider (e.g., Cartesia).
- Seamless Transition: When a unique segment causes a cache miss, the cached audio is emitted. The primary TTS provider, having received the entire sentence (unaware of the cache), generates it with natural prosody. Critically, the segment of audio corresponding to the already-played cached prefix is suppressed from the provider’s output. Only the newly generated audio for the unique part is appended, creating a “seamless” transition that is imperceptible to the user. This gives the illusion of incredibly fast, natural speech, masking the underlying generation latency.
The presentation concludes with a demonstration of a full clinic booking call where these intricate systems work together flawlessly, resulting in a fluid and natural conversational experience for the user, completely hiding the complex architectural optimizations beneath.