How LiveKit Became An AI Company By Accident
Russ d'Sa, CEO of LiveKit, recounts the company's unexpected journey from a pandemic-era open-source WebRTC project to becoming a crucial infrastructure provider for AI voice interfaces, most notably for OpenAI's ChatGPT. He details the serendipitous moments that led to this pivot and shares his vision for LiveKit as the nervous system for a multimodal AI future.
LiveKit began as an open-source project in the middle of the pandemic to address a clear need. The internet’s primary protocol, HTTP (Hypertext Transfer Protocol), was designed for text, not the rich, real-time audio and video streams people suddenly relied on for human connection. The protocol for that, WebRTC, was complex and required significant infrastructure to implement at scale, with only giants like Google Meet, Discord, and Zoom having mastered it.
The initial goal was to build the infrastructure—the “Stripe for real-time audio and video”—so developers wouldn’t have to build it from scratch. This idea stemmed from personal pain while working on a side project. The project quickly found product-market fit with major companies like Spotify, Adobe, and eBay using the open-source version, which validated the need for a commercial, managed cloud offering.
The Serendipitous Pivot to AI
The transition into an AI company was entirely accidental. After launching LiveKit Cloud in late 2022, around the same time ChatGPT was released, I created a demo to voice-enable the chatbot. I jerry-rigged a solution using a headless browser to pipe speech-to-text into the chat interface, thinking it would surely go viral. It didn’t.
Five months later, a pivotal series of events unfolded. A large cloud provider, after a five-hour meeting, presented an ultimatum: be acquired for $20 million, license our technology, or face being crushed by their own open-source alternative. The offer was underwhelming, given we had raised $15 million. They added a crucial qualifier: “If you can somehow tie what you’re doing to GenAI, the offer could be $200 million.” At that moment, I saw no connection.
Terrified by the threat, the focus was simply on moving faster. That same day, driving home, an email arrived with the subject “Hello from OpenAI.” They had found the forgotten voice demo and had been secretly using LiveKit with a personal Gmail account for three weeks to prototype a voice interface for ChatGPT. They loved the platform and wanted to build it for real. This was the moment the pieces connected.
The New Vision: Building the Nervous System for AGI
The collaboration with OpenAI to ship ChatGPT’s voice mode crystallized a new vision for the company. If the goal of AI labs is to build AGI—a synthetic, human-like computer—then interaction with it will be fundamentally human. We interact using our eyes, ears, and mouths. The digital equivalents are cameras, microphones, and speakers.
LiveKit, which was built to connect humans to other humans, could seamlessly be used to connect a human to a human-like machine.
If OpenAI and Anthropic and Gemini… are going to build the brain, LiveKid has an opportunity to build the nervous system to carry signals to and from that brain wherever they originate.
The company is now positioned to be the backbone of multimodal AI. The long-term bet is that as models become more capable, all AI will become multimodal, and LiveKit will provide the core infrastructure.
Building the End-to-End AI Platform
We are now all-in on building the platform for voice, video, and physical AI. The mission is to give any application the ability to see, hear, speak, and, in the case of robotics, move.
The platform is still evolving but is designed to cover the entire lifecycle of building voice agents:
- Transport Network: The original low-latency infrastructure for streaming audio and video.
- Agents Framework: A “Next.js for voice agents” to orchestrate and build voice applications.
- Deployment & Scaling: A hosting product to manage stateful, always-on voice agents, which have different scaling paradigms than traditional web applications.
- Testing & Evaluation: Tooling to ensure agents are performing as intended.
- Observability: A “DataDog for voice AI” to provide visibility into the user experience and system performance.
This end-to-end platform is being built in direct response to community needs, aiming to provide a single foundation to go from idea to a scaled, production-ready business.
The Nuances of Building Human-Like Voice AI
Voice is a fundamentally higher-bandwidth medium than text. It carries rich data through prosody, intonation, and cadence, which can completely change the semantics of what is being said. This richness makes it a natural medium for interaction but also makes it incredibly difficult to create a synthetic, human-like experience.
Achieving this requires getting many details right:
- Conversational Dynamics: Managing interruptions, turn-taking, and low-latency streaming is critical.
- Reliability: The entire chain—from transcription quality to LLM tool-calling reliability—must be robust.
- Latency Perception: If an agent doesn’t respond quickly, it can fall into an “uncanny valley” and be rejected by users. However, the tolerance for latency is highly use-case dependent. For patient intake at a hospital, a 2-second delay is acceptable, mimicking a human looking up information. For immersive language learning, that same delay would feel unnatural and break the experience.
- Feedback During “Thinking”: When an AI needs time to reason or call a tool, silence is deadly. Users assume the system is broken. The solution is to provide immediate feedback, either through non-lexical sounds (“Uh, let me see…”) or explicit statements (“I’m going to do something…”), much like humans do. This primes the user to wait, maintaining the conversational flow.