Models
Currently nvidia/Nemo3-ASR-Streaming is served over realtime sessions. A model that is not refuses the connection withmodel_not_realtime (close code 1008).
Connect
Authorization: Bearer $DEEPINFRA_API_KEY. Browsers, which cannot set headers on a WebSocket, can pass the key as the subprotocol openai-insecure-api-key.<key> instead.
Optional query parameters, for session rotation: going_away=1, going_away_test_s=<10 to 600>, rotate_from=<session id> and preroll_s=<0 to 600>.
Example
Client events
- session.update: sets the input audio format. Send it first. Audio is 16-bit little-endian mono PCM at any sample rate (it is resampled on our side); the default is 24000 Hz.
- input_audio_buffer.append: a base64-encoded chunk of PCM audio. Send audio in real time, for example 100 ms per message; sending audio much faster than real time ends the session.
- session.close: transcribes any open utterance and ends the session.
Server events
- session.created, session.updated: the session is ready, and the format you set.
session.idis the session’s id. A session that opted in to rotation also getspreroll_s(the pre-roll granted, often 0) andalign_samples(in your sample rate, so read it fromsession.updated). - conversation.item.input_audio_transcription.delta: partial text as it is recognized.
deltais the new text;transcriptis the whole open utterance so far, which may revise earlier words. - conversation.item.input_audio_transcription.completed: the final
transcriptof an utterance, and itswords, each withword,startandendin seconds from the session’s first audio sample. A long utterance can arrive as several completed items, and the last item of a session can come withoutwords. - session.going_away: only for a session that opted in; see Session rotation.
- session.end: sent before a normal close, with
stats.audio_secondsandstats.utterances. - error:
error.codesays what went wrong. An error that ends the session is followed by a close.
Close codes
- 1000: the session ended normally, after
session.close. - 1001: a
session.going_awaydeadline passed (going_away). Open a new session. - 1012: the server is restarting (
server_restart), or the session lost its reservation (session_expired). Reconnect. - 1013: try again later, with backoff: for example
server_is_overloaded,websocket_connection_limit_reached,rate_limit_exceeded,idle_timeoutoraudio_backlog. - 1014: the model failed the session (
inference_error). Reconnect. - 1011: an internal error. Retry with backoff, and report it if it repeats.
- 1008, 3000, 3003: these fail again until the request or the account changes (for example
credit_balance_exhausted,model_not_found,invalid_api_key,forbidden).
Session rotation (going_away)
We sometimes have to end a live session, for example to move it off a model server or to restart our gateway. A session that opts in is told first, so it can move to a new session without losing audio.- Opt in with
going_away=1on the connect URL. - When the session must end, it gets session.going_away with
reason(model_pod_drain,gateway_shutdownortest) anddeadline_s, the seconds until it is ended. A second event can follow if the deadline moves sooner. - Before the deadline, open a new session with
rotate_from=<old session id>&preroll_s=<seconds>.preroll_sis how much recent audio you will send again, up to 600; the grant, capped by the model, ispreroll_sin itssession.updated, and is 0 if the old session is not yours, is on another model, was already taken over, or ended more than 60 s ago (an ended session can be taken over only if it had opted in). - On the new session, send at most the last
preroll_sseconds at once, starting at a sample that is a multiple ofalign_samplescounted from the old session’s first sample; then send live audio to both sessions. - Keep the old session until the new one catches up, then join on words: find 3 consecutive words with the same text on both sessions whose start times are within 1 s of each other (the new session’s
wordscount from its first sample, so add the replay’s offset). Use the old session’s words through that run and the new session’s after it, and close the old session. If no such run comes within about 20 s, or the old session closes first, switch anyway at the last word the old session gave; a word there can then be lost or repeated. - At the deadline, a session still open gets the error
going_awayand closes with 1001. If there was too little time to tell it, it closes as a session that did not opt in would: 1012 for a gateway restart, 1014 for a model server. - To test, connect with
going_away_test_s=N(10 to 600): atestevent comes N seconds into the session, and the session is closed with 1001 60 s later, like a real one.
client.realtime.connect(model=MODEL, extra_query={"going_away": "1"}). The session.going_away event arrives in async for event in conn with event.reason and event.deadline_s; match on event.type, as the SDK has no class for it. event.session.preroll_s and event.session.align_samples are on session.updated, and rotate_from, preroll_s and going_away_test_s go in extra_query as well.
Rotation example
This session asks for a testsession.going_away 10 seconds in. On the event, it opens a new session with a 10 s pre-roll, replays that audio, streams live audio to both sessions, joins on words and closes the old session. On a 150 s recording its transcript matched a session without rotation: 343 words each, none lost or repeated.
Python
Billing
Usage is billed per second the session is connected. A session that rotates in also bills the pre-roll it sent ahead of real time. The session it took over is not billed for 60 s from the takeover if it was told by a real event (not atest one), so the overlap is billed once.