Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

Bohan Li of EliseAI unveils a sophisticated architecture for real-time voice agents that merges high intelligence with low latency, akin to a self-driving car's perception, planning, and control systems. Key innovations include a speculative transcriber combining fast streaming with accurate contextual correction, background agents for offloading LLM tool calls and enabling eager generation, and a clever prefix cache for speech synthesis that seamlessly blends pre-cached audio with dynamically generated speech to mask underlying processing delays.