Real-Time Voice with the Realtime API

By Michael Samuel Naeem · August 13, 2025 · 6 min read

Phone support bots that wait three seconds after you stop talking feel broken. Users expect the rhythm of human conversation—overlap, interruption, quick acknowledgments. OpenAI's Realtime API maintains a persistent WebSocket where audio flows in both directions, processed by a single multimodal model. You build a voice agent without stitching together separate STT, chat, and TTS services.

Session lifecycle

Client                    Realtime API (wss)
  |--- session.update ---->|  (configure voice, tools, VAD)
  |--- input_audio_buffer ->|  (stream PCM16 chunks)
  |<-- response.audio -----|  (stream output audio)
  |--- response.cancel --->|  (on user interrupt)
  |--- session end -------->|

Audio format: 24 kHz PCM16 mono, base64-encoded in JSON events. Chunk size: 20–100 ms of audio per message balances latency and overhead.

Server-side session setup

Never expose your API key in browser JavaScript. Proxy through your backend:

# FastAPI endpoint that upgrades to WebSocket proxy
from fastapi import WebSocket
import websockets
import os

@app.websocket("/voice")
async def voice_proxy(client_ws: WebSocket):
    await client_ws.accept()
    async with websockets.connect(
        "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview",
        extra_headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
    ) as api_ws:
        await api_ws.send(json.dumps({
            "type": "session.update",
            "session": {
                "modalities": ["text", "audio"],
                "voice": "alloy",
                "input_audio_format": "pcm16",
                "output_audio_format": "pcm16",
                "turn_detection": {"type": "server_vad"},
                "instructions": "You are a concise support agent for Acme Cloud.",
            },
        }))
        await asyncio.gather(
            relay(client_ws, api_ws),
            relay(api_ws, client_ws),
        )

Client audio capture

const SAMPLE_RATE = 24000;
const ws = new WebSocket("wss://app.example.com/voice");

const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const ctx = new AudioContext({ sampleRate: SAMPLE_RATE });
const source = ctx.createMediaStreamSource(stream);
const processor = ctx.createScriptProcessor(4096, 1, 1);

processor.onaudioprocess = (e) => {
  const pcm = float32ToInt16(e.inputBuffer.getChannelData(0));
  ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: btoa(String.fromCharCode(...new Uint8Array(pcm.buffer))),
  }));
};
source.connect(processor);
processor.connect(ctx.destination);

Commit audio when VAD detects end of speech, or let server VAD handle turn boundaries automatically.

Function calling in voice

{
  "type": "session.update",
  "session": {
    "tools": [{
      "type": "function",
      "name": "check_order_status",
      "description": "Look up order by ID",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": { "type": "string" }
        },
        "required": ["order_id"]
      }
    }],
    "tool_choice": "auto"
  }
}

On response.function_call_arguments.done, execute the function and inject the result:

{
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_abc",
    "output": "{\"status\": \"shipped\", \"eta\": \"2025-08-15\"}"
  }
}

Then send response.create to let the model speak the answer.

Interruption handling

When input_audio_buffer.speech_started fires while audio plays:

ws.send(JSON.stringify({ type: "response.cancel" }));
audioQueue.clear();

On the client, maintain a short playback buffer. Canceling within 300 ms of detected speech sounds natural; longer delays feel like the bot is ignoring the user.

Latency budget

Log response.audio_transcript.done timestamps against input_audio_buffer.speech_stopped to measure your actual turn latency.

Cost and limits

Realtime API bills per audio token (input and output). A 5-minute conversation costs roughly $0.30–$0.60 depending on model. Set max_response_output_tokens to prevent runaway monologues. Monitor session duration and implement idle timeouts (60 seconds of silence closes the session).

WebSocket session management

Production Realtime API requires robust session lifecycle:

class RealtimeSession {
  constructor(apiKey) {
    this.ws = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview",
      ["realtime", `openai-insecure-api-key.${apiKey}`]);
    this.audioQueue = [];
    this.isResponding = false;
  }

  configure() {
    this.ws.send(JSON.stringify({
      type: "session.update",
      session: {
        turn_detection: { type: "server_vad", silence_duration_ms: 500 },
        input_audio_format: "pcm16",
        output_audio_format: "pcm16",
        max_response_output_tokens: 512,
        instructions: "You are a helpful voice assistant. Be concise.",
      }
    }));
  }

  sendAudio(pcm16Buffer) {
    this.ws.send(JSON.stringify({
      type: "input_audio_buffer.append",
      audio: btoa(String.fromCharCode(...new Uint8Array(pcm16Buffer))),
    }));
  }
}

Always configure session before sending audio. Server VAD handles turn detection — don't implement client-side VAD unless latency requires it.

Audio format and buffering

Realtime API expects PCM16 at 24kHz mono:

// Capture from microphone
const audioContext = new AudioContext({ sampleRate: 24000 });
const processor = audioContext.createScriptProcessor(4096, 1, 1);
processor.onaudioprocess = (e) => {
  const pcm16 = float32ToPcm16(e.inputBuffer.getChannelData(0));
  session.sendAudio(pcm16);
};

// Playback: buffer 100ms before starting to avoid underruns
const PLAYBACK_BUFFER_MS = 100;
function onAudioDelta(base64Audio) {
  audioQueue.push(decodePcm16(base64Audio));
  if (!isPlaying && bufferDuration() >= PLAYBACK_BUFFER_MS) startPlayback();
}

Mismatch in sample rate causes pitch/speed distortion. Resample in browser if mic doesn't support 24kHz natively.

Function calling in voice sessions

Extend voice agents with tool use:

// Register tools in session config
session: {
  tools: [{
    type: "function",
    name: "lookup_order",
    description: "Look up order status by order ID",
    parameters: { type: "object", properties: { order_id: { type: "string" } } }
  }]
}

// Handle tool calls
ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type === "response.function_call_arguments.done") {
    const result = await lookupOrder(msg.arguments.order_id);
    ws.send(JSON.stringify({
      type: "conversation.item.create",
      item: { type: "function_call_output", call_id: msg.call_id, output: JSON.stringify(result) }
    }));
    ws.send(JSON.stringify({ type: "response.create" }));
  }
};

Voice + tools enables transactional voice agents (order lookup, appointment booking) — not just conversational.

Failure modes

• No barge-in handling — user speaks while bot talks; feels ignored
• Wrong audio sample rate — pitch distortion; validate 24kHz PCM16
• No idle timeout — session runs indefinitely; cost accumulates
• WebSocket not reconnected on drop — session lost mid-conversation
• max_response_output_tokens not set — runaway monologue; unexpected cost

Production checklist

• Server VAD configured (silence_duration_ms: 500)
• Barge-in: response.cancel on speech_started during playback
• Idle timeout: close session after 60s silence
• Audio format: PCM16 24kHz mono validated end-to-end
• max_response_output_tokens set (512 for concise, 1024 for detailed)
• Session duration and cost monitored per conversation

Resources

• OpenAI Realtime API guide — event reference and session config
• OpenAI Realtime WebSocket protocol — full event type listing
• Web Audio API (MDN) — browser audio capture and playback
• WebRTC VAD alternatives — client-side speech detection
• FastAPI WebSocket docs — proxy pattern for browser clients

Production notes for LLM stacks

When multimodal-realtime-voice-api sits on an inference or RAG path, treat user prompts and retrieved chunks as untrusted input. Log correlation IDs and policy decisions—not raw prompts—in production telemetry. Gate risky operations behind explicit authorization at the gateway, not inside ad-hoc tool handlers.

Roll out changes with shadow mode first: record what would have happened under the new rule without blocking traffic. Compare deny rates, latency impact, and false positives for at least one business week before enforcing. Pair enforcement with a runbook entry: symptom, dashboard, rollback (feature flag or config), and owner.

Load-test with production-shaped concurrency. LLM workloads burst differently from CRUD APIs—tail latency and token throttling dominate. If real-time voice with the realtime api protects an invariant (security, billing, data residency), prove the invariant with an automated test that fails CI when someone removes the check.

What teams get wrong

Teams copy a reference architecture without matching their compliance tier, then discover in audit that logs, backups, or support exports reintroduced the data they thought they had eliminated. Another pattern: shipping the demo integration without idempotency, then fighting duplicate side effects when clients retry on model timeouts.

Document the tradeoff you chose—strictness vs recall, cost vs quality, sync vs async—and the metric that tells you if the choice still holds six months later.