OpenAI’s groundbreaking new model, GPT Realtime is built for real conversations. The Realtime API is now generally available with native image input, SIP phone support, MCP tools, and lower latency and cost. Developers are shipping voice agents that respond quickly, sound human, and integrate with tools, for example, Zillow and T‑Mobile have launched production experiences using the Realtime API.
What Is GPT Realtime? A Simple Explanation
GPT Realtime is a single multimodal model that listens, reasons, and speaks natively. Earlier voice agents chained speech-to-text, an LLM, and text-to-speech, which introduced lag and chopped responses. GPT Realtime replaces that pipeline: you talk, it understands, and it responds in continuous spoken audio.
The model accepts multiple input types in one session: voice, typed text, and images. It generates expressive audio with natural intonation, pacing, and emotion, and it can switch languages mid‑sentence.
Why “Realtime” Matters
Conversation depends on rhythm. Long pauses break the flow. GPT Realtime is optimized for low latency and natural turn-taking: it can detect interruptions, laughters, or sighs, and pause mid-sentence when the user jumps in—behaving more like a human interlocutor than a demo.
How GPT Realtime and the Realtime API Work (Plain English)
Think of the Realtime API as a direct connection between your app and a voice that understands, reasons, and speaks. The flow is:
1) Your microphone captures sound and converts it to digital audio samples.
2) Your app streams small chunks of that audio to the Realtime API.
3) The model converts audio into internal tokens for reasoning, decides what to say, and begins speaking back while still processing.
4) Audio streams back to the client as it’s generated, so responses arrive before a full sentence is complete.
The system is multimodal: you can send text messages alongside audio (for example, a system instruction), attach images to a session, and allow the model to call tools or functions mid-conversation. SIP support connects Realtime sessions to traditional phone systems. For complex integrations, you can point the session at an MCP (Model Context Protocol) server to give the model controlled access to tools and data without hard-wiring each integration.
Quick mental model: You speak → audio chunks → GPT Realtime (understands and plans) → optional tool/API calls → streams natural speech back.
Sounding Natural: Voices, Tone, and Languages
The speech generator produces natural variations in tone, emphasis, and pace. You control style via system messages (for example:
"Be concise and confirm numbers slowly.")
The model supports mid-sentence language switches and can follow scripts precisely for compliance use cases. New voices called Marin and Cedar are available and existing voices have been upgraded.
Smarter Conversations: Instructions, Tools, and Context
GPT Realtime is tuned for real-world tasks like customer support, tutoring, and onboarding. It follows persistent rules (for example, policy guardrails), calls functions with appropriate arguments (e.g., check order status), and can continue the conversation while long-running tools execute (asynchronous function calling). It repeats alphanumerics accurately and confirms critical details back to the user.
Visual Input: Add Images to Ground the Conversation
You can attach photos or screenshots to a session and ask the model about them. Examples include asking whether a wiring setup is safe, reading and summarizing text in a screenshot, or identifying a part number in an image. The model blends visual and audio information and can answer instantly in voice.
Plain‑English Guide to Key Terms
GPT Realtime: A speech-to-speech multimodal model that understands and generates spoken language in real time.
API: A software interface to connect your app to the model.
Multimodal: The model can process voice, text, and images together.
LLM: A large language model trained on vast text corpora.
Latency: The delay between user input and the system response; low latency feels natural.
Alphanumerics: Combinations of letters and numbers (e.g., AB12C3).
System Message: Initial instructions that define tone and rules for the assistant.
Asynchronous Function Calling: Letting the assistant keep speaking while long tasks (like database lookups) finish.
SIP: Session Initiation Protocol for internet voice/video calls and PBX integration.
MCP: Model Context Protocol to grant controlled access to tools and data.
Audio Tokens: Small units of sound the model processes; more tokens mean longer audio.
Cached Input Tokens: Previously seen input that can be reused cheaply.
Classifiers: Filters that detect harmful or disallowed content.
CSAT: Customer satisfaction score.
If you want background reads, see the following resources: Speech recognition overview, Natural language processing, and SIP explained.
Try It in Minutes: The Playground
The quickest way to experience GPT Realtime is via the OpenAI Playground. Steps to try it:
1) Open the Playground, choose Audio > Realtime, and select the GPT Realtime model.
2) Allow microphone access and speak.
3) Interrupt it mid-sentence, change tone on the fly, or try multilingual prompts.
4) Add a system message to modify behavior, for example: "Be concise and friendly. Confirm numbers slowly. If I interrupt, pause and ask a short follow-up."
Helpful links: OpenAI Docs, OpenAI Playground, OpenAI Pricing.
Build Your First Voice Agent (A Minimal Plan)
Start simple: pick one task, one voice, and one success metric (for example, “check order status by email or phone number and read the result back clearly”).
High-level flow:
1) Create a session with a clear system message defining tone and policies.
2) Stream microphone audio to the session in small chunks.
3) Play the model’s audio as it arrives.
4) When the model calls a tool (e.g., lookupOrder), run your function and send results back.
5) Keep the loop until the user ends the session.
Start in the Playground, then move to the SDK to wire your function. After stabilizing tool calls, add barge-in (interrupt) handling and test with real users.
Connecting to Your Stack: Tools, SIP, and MCP
Tool calling: define a few safe functions such as get_customer, check_order, or book_appointment. The assistant decides when to call them; you validate inputs, return results, and the assistant speaks outcomes naturally.
SIP phone support: route real phone calls to your agent for support or sales via PBX systems.
MCP (Model Context Protocol): point your Realtime session at an MCP server to provide controlled access to tools and data without one-off integrations.
Refer to the official docs for implementation: OpenAI Realtime Docs and Pricing and limits.
How GPT Realtime “Understands” Audio (Beginner Friendly)
Your microphone captures sound waves as digital samples. The model ingests these samples in small chunks; neural networks convert audio into compact representations (audio tokens). The language component reasons over those tokens and plans responses. A speech generator converts planned text into audio tokens and streams them back, enabling you to hear responses before the model completes full sentences.
Images are processed as pixel grids; the model extracts features and blends them with audio/text context to answer questions about what it sees.
Prompting and Conversation Design That Work
Design tips:
• Start with a style instruction like "Be concise, empathetic, and confirm key numbers slowly."
• Set guardrails: for example "Never process refunds over $10. If asked, escalate."
• Provide one or two example turns to teach tone.
• Keep responses short and invite follow-ups.
• Confirm critical info by repeating it back.
• While a tool runs, have the assistant keep small talk focused and minimal.
Pricing, Limits, and Cost Control
Launch pricing (announced): input audio tokens at $32 per 1M, output audio tokens at $64 per 1M, and cached input tokens at $0.40 per 1M. To control costs, favor concise responses, summarize or truncate long context, cache repeated prompts, and monitor average tokens per call and cost per successful outcome.
Safety, Privacy, and Responsible AI
OpenAI provides safety systems and classifiers to flag or halt harmful content. Use system messages to enforce policies and scripts. Enterprise commitments cover privacy, and EU data residency is supported for regulated teams. Keep API keys secure and validate tool inputs/outputs on your side to minimize misuse.
Real‑World Inspirations
Examples of deployed use cases include Zillow guiding multi-step home searches and T‑Mobile building a device-upgrade assistant quickly. Lessons from teams shipping voice agents include keeping instructions tight, making the agent interruptible, confirming alphanumerics and sensitive details, and starting with one intent before expanding.
Key Facts at a Glance
| Model | GPT Realtime (speech-to-speech, multimodal) |
| New features | Image input, SIP calling, MCP server support, reusable prompts |
| Voices | New: Marin and Cedar; existing voices upgraded |
| Instruction following (audio eval) | 30.5% on MultiChallenge |
| Function calling (audio eval) | 66.5% on ComplexFuncBench |
| Reasoning (audio eval) | 82.8% on Big Bench Audio |
| Pricing | $32/1M audio input tokens, $64/1M audio output tokens; cached input $0.40/1M |
Your Next Steps
Try GPT Realtime in the Playground, spin up a focused pilot with one task and one metric, wire a single tool (e.g., order lookup), and iterate weekly: refine system prompts, shorten responses, tighten guardrails, and monitor cost per outcome. If you have an OpenAI API key, open the Playground, switch to gpt-realtime, and start talking—then stand up a small pilot in your app.
What is GPT Realtime?
GPT Realtime is a multimodal speech-to-speech model that listens, reasons, and speaks directly in audio. It streams input and output with low latency via the Realtime API for live voice conversations.
What new features come with the Realtime API?
The Realtime API adds image input, SIP phone support, MCP integration, streaming audio input/output, and new voices (Marin and Cedar) along with upgrades to existing voices.
How is GPT Realtime different from traditional voice AI pipelines?
Traditional systems used separate components for speech-to-text, an LLM, and text-to-speech. GPT Realtime uses one multimodal model to handle listening, reasoning, and speaking in a single continuous flow.
How does streaming audio input/output work?
You send small audio chunks continuously and receive generated speech as it’s produced, which makes interactions feel live and reduces wait times.
What are the new voices and how can I control tone?
The new voices are Marin and Cedar, and existing voices are upgraded. Control tone and style through system messages or prompts (for example, specify “calm and professional” or “excited but concise”).
Can Realtime handle images in conversations?
Yes. You can attach images to a session, and the model will describe or answer questions about them based on the visual content.
What kinds of tool integrations does Realtime support?
Realtime supports tool/function calling for actions like CRM lookups, SIP for phone network connectivity, and MCP to provide the model controlled access to tools and data for complex workflows.
How do I start using Realtime in the Playground?
Create an OpenAI account and API key, open the Playground, go to Audio > Realtime, select gpt-realtime, enable your microphone, and start talking. You can interrupt, change tone, or try multilingual prompts.
How is pricing structured for Realtime?
Pricing is $32 per 1M audio input tokens and $64 per 1M audio output tokens, with cached input tokens at $0.40 per 1M. Costs scale with usage and can be managed through token limits and concise responses.
What safety and privacy considerations should I know?
OpenAI provides safety systems and classifiers to mitigate harmful content. Enterprise privacy commitments and EU data residency options are available. Use system messages to enforce policies and validate tool inputs/outputs to prevent misuse.