Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Kramer (Daily)
Sean DuBois (OpenAI, Pion) and Kwindla Hultman Kramer (Daily, Pipecat) argue that to build successful real-time AI applications, developers must start from the network layer up, prioritizing WebRTC over WebSockets to manage latency effectively and enable advanced features like interruption and state management.
For a conversational voice experience to feel natural and engaging, latency is the most critical factor. While many aspects of building voice agents are similar to other multi-turn AI agents, the entire application must be engineered from the ground up for fast response times. Humans expect response latencies of around 500 milliseconds in conversation, and anything significantly over a second leads to poor user experiences and high drop-off rates. This “voice-to-voice” latency—the time from when a user stops speaking to when they hear the first byte of audio back from the AI—is a budget that gets consumed by multiple components, including audio buffering, transcription, LLM inference, and text-to-speech (TTS).
The Core Mistake: WebSockets vs. WebRTC
The single biggest mistake developers make when building real-time voice applications is choosing the wrong networking protocol. Many developers default to WebSockets for long-lived connections, which is a critical error for real-time audio and video.
- WebSockets: Built on TCP, they guarantee in-order, reliable delivery of every data packet. If a single packet is lost or delayed, the entire stream is blocked until that packet is re-transmitted and received. This is ideal for web requests or structured data, but disastrous for real-time audio, where it introduces significant glitchiness, high latency, and unexpected disconnections in 10-15% of real-world network conditions.
- WebRTC: Designed specifically for real-time media. It uses UDP-based protocols and sophisticated buffer management to handle packet loss and network jitter gracefully. It prioritizes the continuous flow of media over perfect delivery, ignoring packets that don’t arrive within the tight latency budget. Think of it as “super fast best effort networking.”
You cannot replicate WebRTC’s performance on top of a TCP-based stack like WebSockets because the operating system’s networking layer will always try to guarantee delivery, blocking everything else.
Beyond Packet Handling: The Built-in Power of WebRTC
WebRTC provides a comprehensive suite of tools essential for high-quality real-time media, which developers would otherwise have to build themselves:
- Audio Processing: Handles complex tasks like audio resampling and packetization.
- Bandwidth Management: Constantly performs bandwidth estimation to adapt the audio bitrate to fluctuating network conditions.
- Observability: Offers standard APIs for statistics and monitoring, crucial for debugging and optimization.
By using WebRTC, developers can focus on application logic rather than wrestling with the low-level complexities of real-time media transport.
The Future is Voice: A New UI Paradigm
WebRTC is already the de-facto standard for real-time communication on the internet, powering applications like a company that was doing surgery over the internet, Facebook Messenger, Zoom, and Discord. Its power and flexibility make it the ideal foundation for the next generation of user interfaces, which will be built on voice.
The current moment in voice AI is analogous to late 2007 in the mobile revolution—we have the first iPhones, but we haven’t invented “pull to refresh” yet. Voice allows us to interact with computers in situations where our hands are not free, unlocking new potential for productivity and assistance.
The flexibility of WebRTC allows for various connection architectures:
- Peer-to-Peer (P2P): Devices can connect directly over a local network, as demonstrated with the “Squabbert” Raspberry Pi demo, enabling serverless, low-latency interactions.
- Client-to-Server: A device can connect to a server in the cloud for more intensive AI processing.
- Multi-Party: Multiple clients and LLMs can be brought together in a single session, enabling use cases like AI agents in meetings.
The ultimate goal is to make this powerful technology accessible to everyone, not just seasoned programmers. The speakers highlighted a project by a non-technical founder, Yashin, who built a bilingual education tool for children with guidance from the community. This showcases the potential for innovation when the barriers to entry are lowered, empowering anyone with a great idea to build the future of voice AI.