Skip to main content
Realtime transcription streams audio to a model over a WebSocket and returns the transcript while the speaker is still talking: partial text as words are recognized, and a final transcript with word timings for each utterance. The protocol is compatible with OpenAI’s Realtime transcription API, so the OpenAI SDKs work with it. For transcribing a finished audio file over HTTP, see Speech Recognition.

Models

Currently nvidia/Nemo3-ASR-Streaming is served over realtime sessions. A model that is not refuses the connection with model_not_realtime (close code 1008).

Connect

Authenticate with Authorization: Bearer $DEEPINFRA_API_KEY. Browsers, which cannot set headers on a WebSocket, can pass the key as the subprotocol openai-insecure-api-key.<key> instead.
Do not ship a long-lived API key to a browser: create a short-lived token limited to this model on your server with POST /v1/scoped-jwt and pass that instead. A subprotocol cannot contain :, so send the returned jwt:<token> as jwt.<token>, as in the JavaScript example below.
Optional query parameters, for session rotation: going_away=1, going_away_test_s=<10 to 600>, rotate_from=<session id> and preroll_s=<0 to 600>.

Example

Client events

  • session.update: sets the input audio format. Send it first. Audio is 16-bit little-endian mono PCM at any sample rate (it is resampled on our side); the default is 24000 Hz.
  • input_audio_buffer.append: a base64-encoded chunk of PCM audio. Send audio in real time, for example 100 ms per message; sending audio much faster than real time ends the session.
  • session.close: transcribes any open utterance and ends the session.

Server events

  • session.created, session.updated: the session is ready, and the format you set. session.id is the session’s id. A session that opted in to rotation also gets preroll_s (the pre-roll granted, often 0) and align_samples (in your sample rate, so read it from session.updated).
  • conversation.item.input_audio_transcription.delta: partial text as it is recognized. delta is the new text; transcript is the whole open utterance so far, which may revise earlier words.
  • conversation.item.input_audio_transcription.completed: the final transcript of an utterance, and its words, each with word, start and end in seconds from the session’s first audio sample. A long utterance can arrive as several completed items, and the last item of a session can come without words.
  • session.going_away: only for a session that opted in; see Session rotation.
  • session.end: sent before a normal close, with stats.audio_seconds and stats.utterances.
  • error: error.code says what went wrong. An error that ends the session is followed by a close.

Close codes

  • 1000: the session ended normally, after session.close.
  • 1001: a session.going_away deadline passed (going_away). Open a new session.
  • 1012: the server is restarting (server_restart), or the session lost its reservation (session_expired). Reconnect.
  • 1013: try again later, with backoff: for example server_is_overloaded, websocket_connection_limit_reached, rate_limit_exceeded, idle_timeout or audio_backlog.
  • 1014: the model failed the session (inference_error). Reconnect.
  • 1011: an internal error. Retry with backoff, and report it if it repeats.
  • 1008, 3000, 3003: these fail again until the request or the account changes (for example credit_balance_exhausted, model_not_found, invalid_api_key, forbidden).

Session rotation (going_away)

We sometimes have to end a live session, for example to move it off a model server or to restart our gateway. A session that opts in is told first, so it can move to a new session without losing audio.
  • Opt in with going_away=1 on the connect URL.
  • When the session must end, it gets session.going_away with reason (model_pod_drain, gateway_shutdown or test) and deadline_s, the seconds until it is ended. A second event can follow if the deadline moves sooner.
  • Before the deadline, open a new session with rotate_from=<old session id>&preroll_s=<seconds>. preroll_s is how much recent audio you will send again, up to 600; the grant, capped by the model, is preroll_s in its session.updated, and is 0 if the old session is not yours, is on another model, was already taken over, or ended more than 60 s ago (an ended session can be taken over only if it had opted in).
  • On the new session, send at most the last preroll_s seconds at once, starting at a sample that is a multiple of align_samples counted from the old session’s first sample; then send live audio to both sessions.
  • Keep the old session until the new one catches up, then join on words: find 3 consecutive words with the same text on both sessions whose start times are within 1 s of each other (the new session’s words count from its first sample, so add the replay’s offset). Use the old session’s words through that run and the new session’s after it, and close the old session. If no such run comes within about 20 s, or the old session closes first, switch anyway at the last word the old session gave; a word there can then be lost or repeated.
  • At the deadline, a session still open gets the error going_away and closes with 1001. If there was too little time to tell it, it closes as a session that did not opt in would: 1012 for a gateway restart, 1014 for a model server.
  • To test, connect with going_away_test_s=N (10 to 600): a test event comes N seconds into the session, and the session is closed with 1001 60 s later, like a real one.
With the OpenAI SDK, opt in with client.realtime.connect(model=MODEL, extra_query={"going_away": "1"}). The session.going_away event arrives in async for event in conn with event.reason and event.deadline_s; match on event.type, as the SDK has no class for it. event.session.preroll_s and event.session.align_samples are on session.updated, and rotate_from, preroll_s and going_away_test_s go in extra_query as well.

Rotation example

This session asks for a test session.going_away 10 seconds in. On the event, it opens a new session with a 10 s pre-roll, replays that audio, streams live audio to both sessions, joins on words and closes the old session. On a 150 s recording its transcript matched a session without rotation: 343 words each, none lost or repeated.
Python

Billing

Usage is billed per second the session is connected. A session that rotates in also bills the pre-roll it sent ahead of real time. The session it took over is not billed for 60 s from the takeover if it was told by a real event (not a test one), so the overlap is billed once.