Introducing gpt-realtime in the API

A significant step has been taken towards enabling human-level voice interaction with AI agents through the general availability of an improved Real-Time API and the release of a new, advanced speech model, GPT-realtime.

GPT-realtime: A Natively Multimodal Speech-to-Speech Model

The new GPT-realtime model represents a shift from classic architectures that chain separate transcription, language, and voice synthesis models. As a single, integrated speech-to-speech model, it natively understands and produces audio, offering several architectural advantages:

• Speed and Low Latency: The unified model reduces processing overhead.
• Native Audio Understanding: It can interpret non-verbal cues like laughter or sighs, leading to more natural interaction.
• Emotional Range: The model demonstrates a wide range of emotions in its speech output.
• Seamless Language Switching: It can switch between languages mid-sentence, as demonstrated in a poem delivered in English, Spanish, and Japanese.

The model was trained in close collaboration with customers to align its performance with production use cases like customer support and academic tutoring.

Core Capabilities and Demonstrations

The model’s advanced capabilities were showcased through several key demonstrations:

• Instruction Following: The model strictly adheres to developer-defined system instructions, even under user pressure. In a demo, it correctly refused to process a refund request exceeding a pre-set $10 limit, politely maintaining the policy despite repeated requests.
• Image Input: For the first time, the Real-Time API supports image input, allowing the model to “see” what the user sees. In a demonstration, the model accurately described fine details from a photo of a child, including a wooden toy train, a small hair clip, and a unicorn toy’s rainbow mane, and even offered thoughtful parental advice based on the visual context.
• Emotional and Tonal Range: The model can adopt different personas and emotional states on command, from the despair of losing a winning lottery ticket to the excitement of finding it.

Training, Performance, and Benchmarks

The model’s improvements are a result of advanced post-training methods and a focus on data quality.

• Training Methodology:

  • The model was trained using a combination of high-quality voice data and specialized reward models to enhance naturalness.
  • A highly sample-efficient reinforcement learning algorithm was employed.
  • A “data flywheel” was established to train the model directly on real customer use cases, improving its performance in practical scenarios.
  • Significant investment was made in data quality, including filtering for speech-related data.

• Performance Gains:

  • Instruction Following: The new model achieves over 30% accuracy on an audio version of the challenging multi-turn instruction following benchmark, a significant increase over previous models.
  • Function Calling: On the complex phone-bench audio evaluation, designed for difficult function-calling scenarios, the model scores 66% accuracy, showing steady improvement.

Targeted improvements were also made based on user feedback, such as better handling of long alphanumeric strings (e.g., VINs, phone numbers) and more reliable behavior when it cannot hear the user clearly.

Real-Time API: Now Generally Available with New Features

The Real-Time API, the platform for building low-latency voice applications, is now generally available and has been enhanced for latency, reliability, and scale. New features include:

• Image Input: As demonstrated, vision capabilities are now integrated.
• SIP Telephony: Eases the development of applications for voice-over-phone use cases.
• MCP (Multi-turn Conversation Protocol) Support: Allows for pluggable capabilities, enabling the model to take actions on the user’s behalf in a natural, conversational way.
• EU Data Residency: Addresses data sovereignty requirements.
• Asynchronous Function Calling: Provides more flexibility for developers.
• Enhanced Context Management: Offers more tools for managing conversation history in a cache-friendly manner.

Enterprise Application: The T-Mobile Use Case

T-Mobile showcased how the new model and API are being used to reinvent their device upgrade process—a common but often complex customer journey. Their AI assistant can now handle nuanced, multi-turn conversations, guiding users through phone selection, checking plan compatibility for services like satellite texting, and answering specific questions.

Srini Gopalan, COO of T-Mobile, emphasized that the new model is “so much more human” and can follow the “random walk” of a customer’s questions. He shared key strategic insights from their AI implementation:

1. Reinvent, Don’t Iterate: Use AI to “smash your existing processes” and rebuild them from the ground up, rather than making incremental improvements like a “10% better IVR.”
2. Align with Brand: Ensure the AI’s application and persona are consistent with the company’s brand and culture. For T-Mobile, this means using AI to “put an expert in your pocket” and eliminate the trade-off between automated service and expert assistance.

• Tags:
• AI Agents
• Speech-to-Speech
• Multimodal AI
• Voice AI
• Real-Time API
• GPT-Realtime
• Function Calling