Build Hour: GPT-Realtime-2

Explore GPT-Realtime-2, OpenAI's advanced voice AI model, through practical demos and a deep dive with Sierra on building production-grade, low-latency voice agents with complex reasoning and tool use.

May 13, 2026 5 min read

Build Hour: GPT-Realtime-2 — source: https://www.youtube.com/watch?v=qGS9Ghnq1RU

OpenAI’s latest advancements in voice intelligence introduce three new models designed to power the next wave of real-time, hands-free applications. These models shift the paradigm from simple voice commands to complex, voice-driven workflows with sophisticated reasoning and tool integration.

Overview of New Realtime Voice Models

Three new models form the foundation of this release, each serving a distinct purpose in the voice AI stack:

GPT-Realtime-Whisper: A streaming speech-to-text model offering tunable latency as low as 200ms. It supports 80 input languages and is ideal for use cases like live captioning, meeting notes, and providing ambient context to agents, enabling earlier function calling and better instruction following.
GPT-Realtime-Translate: Powers live, low-latency conversations by translating over 70 input languages into 13 output languages. It’s designed to make language barriers disappear in applications like video calls, live streams, and global customer service.
GPT-Realtime-2: The most intelligent of the new models, bringing GPT-4 class reasoning to voice interactions. It excels at prompt adherence, multilingual performance, and complex tool calling, which are critical for production voice workflows. This model enables three primary development patterns:
- Voice-to-Action: For hands-free, voice-driven applications.
- Systems-to-Voice: For AI assistants that organize and present information through spoken voice.
- Voice-to-Voice: For natural, human-like conversational experiences like customer service calls.

Key Capabilities of GPT-Realtime-2

GPT-Realtime-2 introduces significant improvements that address major requests from the developer community:

Preambles: The model can use conversational fillers like “Let me check on that…” before beginning its reasoning process, making interactions feel more natural and human.
128K Context Window: A 4x increase in the context window, equivalent to roughly one hour of conversation, allows for better instruction following and higher intelligence by reducing the need for truncation.
Parallel Tool Calling: The model can execute multiple tool calls simultaneously, moving beyond the previous waterfall approach and enabling more complex, efficient actions.
Improved Domain Understanding: Enhanced comprehension of specialized vocabularies in fields like healthcare and AI.
Controllable Expressiveness: Developers can instruct the model to adopt different tones, such as whispering or showing excitement.
Dynamic Tone Matching: The voice-to-voice capability matches the user’s emotional tone for more empathetic and natural-sounding interactions.

Demo 1: Voice-Powered E-commerce Agent

The first demonstration showcased a voice agent integrated into an e-commerce website. This highlighted the “voice-to-action” pattern where the agent wasn’t just chatting but actively operating the user interface.

Contextual Awareness: The agent recalled the user’s previous shopping activity and identified remaining items on their list (a tent and hiking shoes).
Complex Tool Use: The agent filtered products based on multi-faceted user requests (e.g., “tents under $450 that fit three to four people”). It also accessed external tools, like a weather API, to provide a forecast for Seattle and help the user make a more informed purchasing decision about the tent’s storm-readiness.
UI Operation: The agent directly manipulated the UI by highlighting products, adding items to the cart, and navigating to the cart page, creating a seamless, hands-free shopping experience. This demonstrated the model’s ability to reason across more than 15 available tools and update the visual experience in real-time.

Demo 2: Product Analytics Dashboard

The second demo illustrated a different “voice-to-action” workflow, positioning the AI as an “analyst in the loop” for a product manager. A key feature was the model’s ability to follow instructions precisely and remain silent unless explicitly asked to speak.

Instruction Following: The user issued a series of voice commands to filter a complex dashboard (filter by Europe, look at the last seven days). The model executed these actions without verbal confirmation, allowing for a rapid, fluid workflow.
Proactive Suggestions: When prompted, the model suggested relevant filters to continue the investigation (e.g., “voice search,” “first-time shoppers”).
Data Analysis and Summarization: The model conducted a root cause analysis by comparing mobile Safari and Chrome data. When asked, it provided a concise, two-sentence verbal summary of its findings, ready for the product manager to share with their engineering team.

The investigation shows a mobile Safari-specific regression where the product detail page size selector validation doesn’t update correctly. So, first-time Europe footwear shoppers get stuck after choosing a size and can’t add to cart.

Customer Spotlight: Sierra’s Production Voice Agents

The team from Sierra, which builds AI customer service agents for large enterprises, shared their experience integrating GPT-Realtime-2 into their production systems.

The Challenge of Production Voice

Operating at enterprise scale presents unique challenges beyond just sounding natural. An agent that violates a business policy or takes a wrong action even 0.1% of the time is not shippable. The core challenge is building agents that businesses can trust to interact directly with their customers.

Sierra’s Agent Harness

While GPT-Realtime-2 provides a powerful foundation, Sierra runs it within a proprietary “agent harness” that adds the necessary infrastructure for reliable and secure production deployment.

Custom VAD: Sierra uses custom-tuned Voice Activity Detection (VAD) models to better handle messy, real-world audio with background noise, accents, and interruptions, giving them more control over turn-taking.
Guardrails and Grounding: The harness defines customer-specific workflows, tools, branding, and policies to keep the agent aligned and prevent errors.
Production Infrastructure: It handles tracing, redaction of sensitive information, PCI-compliant payment flows, and other features to ensure agents are accurate, safe, and observable.

Performance and Evaluation

Sierra’s initial testing with GPT-Realtime-2 showed significant performance gains:

Latency: Calls were roughly 30% faster at P50 and up to 200% faster at P90 compared to their previous cascaded (STT -> LLM -> TTS) system.
Evaluation: Sierra emphasizes end-to-end task success over just voice quality. They use simulations that replay realistic customer calls to measure whether the agent successfully completes its objective, as that is the “real bar for production voice.”

Common Failure Modes in Voice AI

The Sierra team highlighted that production calls are rarely ideal. Agents must handle:

Interruptions and Messy Audio: Customers calling from noisy environments (e.g., on a highway, with kids in the background).
High-Stakes Errors: Misunderstanding a name spelling can lead to failed database lookups; canceling the wrong flight can have severe consequences.
Backchanneling: Human conversations are filled with “mhms” and “uh-huhs.” Agents must learn to distinguish ignorable backchannels from substantive responses, a task where cascaded models traditionally had an edge but voice-to-voice models are rapidly improving.