Build Hour: GPT-Realtime-2

TL;DR: OpenAI announced GPT Realtime-2 and two companion models during Build Hour, showcasing smarter, more natural voice interactions with 128k context, parallel tool calls, and dynamic voice cloning. Two demos—a shopping assistant and an analytics dashboard—illustrated production-grade "voice-to-action" workflows. ## Release Overview: Three New Models Last Thursday, OpenAI released audio updates comprising three models, each targeting different voice workflows: - **Real-time Translation Model**: Supports 70+ input languages and 13 output languages, offering low-latency streaming translation. In the demo, Sarah and Terry conversed in English while Spanish translation appeared in real-time on the right. This model bridges language barriers for voice conversations, making it suitable for video calls, live streaming, and customer service. - **GPT Realtime Whisper Model**: Streaming speech recognition with latency as low as 200ms, supporting 80 input languages. It triggers function calls earlier, follows instructions better, and is ideal for subtitles, meeting notes, and ambient agent scenarios. - **GPT Realtime 2**: The most intelligent voice model, bringing GPT-5-level reasoning into voice. It shows significant improvements in instruction following, tool calling, and multilingual performance. These three models map to three building approaches: - **Voice to Action**: Hands-free, voice-driven applications - **System to Voice**: Orally organized voice "chief of staff" - **Voice to Voice**: Customer service calls, e.g., T-Mobile's global deployment ## Key Feature Improvements - **Dynamic Voice Cloning & Emotion Matching**: Can distinguish multiple speakers and adapt tone based on context. Also supports an "opening line" feature, allowing the model to say "Let me check that" before reasoning—more like a real human conversation. - **Context Window Increased to 128k**: Holds approximately one hour of conversation, preventing truncation-related information loss, significantly boosting instruction following and intelligence. - **Parallel Tool Calls**: No longer requires sequential tool invocations; the model can initiate multiple tool operations simultaneously. - **Controllable Expressiveness**: Developers can guide the model to whisper, sound excited, be jealous, etc. - **Better Domain Vocabulary Understanding**: Improved recognition of specialized terms in fields like healthcare and AI. - **Context Overrides**: Supports more complex agent behaviors. ## Demo 1: Voice-Driven Search Agent Erica demonstrated a shopping assistant named "Supply Co," an outdoor gear e-commerce site. She interacted with the model via voice: 1. **Wake the Assistant**: Asked it to recall last shopping progress (tool: read shopping plan). 2. **Find a Tent**: Specified budget and number of people (tool: search products and filter by price). 3. **Check Bad Reviews**: Asked to view one- and two-star reviews (tool: read reviews and summarize). 4. **Check Weather**: Requested the upcoming two-week weather forecast for the travel destination (tool: call external weather API). 5. **Synthesize Recommendations**: The model assessed tent suitability based on weather and recommended add-ons (ground cloth, stakes). 6. **Add to Cart**: Added the tent and boots to the cart, displaying the total price. Key points: - The model selected from 15–20 tools, calling multiple ones in parallel (search, read reviews, weather API). - Voice interaction felt natural and synced with UI visual updates. - Erica emphasized this was not "one voice input, one action output" but a true multi-step reasoning shopping assistant. ## Demo 2: Product Analytics Dashboard Erica switched to a product manager perspective, using the AI assistant to analyze conversion issues in the European market. 1. **Filter Data**: By European region, comparing the past 7 days vs. the previous 7 days. 2. **Spot Anomaly**: The model highlighted a data drop on Safari browser. 3. **Deep Dive**: Further filtered by "first-time shoppers," "footwear," and "voice search," revealing significant activation churn. 4. **Root Cause Analysis**: Compared mobile Safari to Chrome; the model initiated root cause investigation, generating dashboards and artifacts. 5. **Verbal Summary**: The model summarized in two sentences: "This is a mobile Safari-specific regression where the size selector validation on the product detail page didn't update correctly, causing first-time European footwear shoppers to get stuck." Key points: - The model spoke only when the user requested; otherwise it just performed actions (filtering, highlighting), demonstrating excellent instruction following. - Users could inspect reasoning traces and conversation history at any time, and text interaction was also supported. - The model effectively wrote code, created charts, and generated artifacts, acting as a "human-in-the-loop analyst" for real-time conversational analytics workflows. ## Voice Application Examples Terry discussed the broad potential of voice interfaces and generated sample images using GPT Imagination: - Smart device integration (e.g., smart home) - Coding assistants (voice-driven programming experience) - Mobile apps (adding voice interfaces to any app) - Voice-controlled games (e.g., motion-based games with up-down jumps) - Coaches: public speaking coach, note-taking assistant - Financial voice agents, voice video calls These examples suggest that future children may converse with AI more often than they type. OpenAI is ready to support this wave. ## Summary & Expectations This release marks a new stage for voice AI: from simple speech recognition and response to intelligent agents that can reason, call tools, and control UIs. Developers now have three distinct models to choose from based on their application needs. The Sierra team will later share expert tips for building voice agents, followed by a Q&A session. --- Source: Build Hour: GPT-Realtime-2 - OpenAI YouTube (https://www.youtube.com/watch?v=qGS9Ghnq1RU)