A realtime transcript arrives twice: first as a draft that keeps changing, then as a final line that does not. Render the draft as a replaceable tail and the final line as history, and the rest of a live caption view is plumbing.
A realtime session is the smallest thing the VocalStack API does. Your server makes one HTTP call and gets back a signed streamUrl. Your client opens a WebSocket to that URL, sends raw audio as it is captured, and reads the transcript back as JSON text frames moments behind the speaker. When the audio ends you send eof, the server flushes, reports what the session measured, and closes.
Nothing is stored. The transcript exists only on the wire, which is what a caption view, a meeting overlay or a voice feature in your own app wants. If you want the words kept, shared and exported afterwards, you want a live transcription and a broadcast instead; the last section points there.
The shape is the one most streaming speech APIs share: a credential exchange over HTTP, a long-lived socket with audio going in and JSON coming out, and some way of telling provisional text from final text. The names differ from one API to the next; the moving parts rarely do. Here, in order:
POST /v1/realtime/sessions with your API key, on your server. The answer is { sessionId, streamUrl, expiresAt }.streamUrl.session.loading heartbeats until session.ready.transcript.draft (replace the tail) and transcript.committed (append a final line).{"type":"eof"}; read session.done.Only step 1 needs your key, so it happens on your server.
Create a key under Developers, API keys with the Write permission. A key is for servers and scripts: the API refuses one sent from a web page or in a URL, so you create the session behind a route of your own, and the page only ever sees the streamUrl.
The URL carries its own token because of a browser limitation that is worth knowing in general. The browser WebSocket constructor cannot set request headers, so there is no Authorization header to put a key in; every browser-facing speech API authenticates its socket with a short-lived token in the URL (or smuggled into a subprotocol name). Short-lived is the point: a URL ends up in logs and history, so what it carries must expire. Connect before expiresAt.
The SDKs are downloaded from the Developers area, not from a package registry; the Install section of the quickstart shows this environment's own commands. With the TypeScript SDK, a server route looks like this:
// On your server, behind a route of yours (POST /api/transcribe, say):
import { VocalStack } from '@vocalstack/sdk';
const vs = new VocalStack({ baseUrl: '{{api}}', apiKey: process.env.VOCALSTACK_API_KEY });
export async function createStream(): Promise<{ sessionId: string; streamUrl: string }> {
const { sessionId, streamUrl } = await vs.realtime.createSession({ spokenLanguage: 'en' });
return { sessionId, streamUrl };
}
Without an SDK it is one fetch. This one also asks for Spanish and turns sentence assembly off, matching the wire sample further down:
const res = await fetch('{{api}}/v1/realtime/sessions', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.VOCALSTACK_API_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ spokenLanguage: 'en', translate: ['es'], sentences: { enabled: false } }),
});
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); // the error envelope: { code, message, … }
const { streamUrl } = (await res.json()) as { sessionId: string; streamUrl: string; expiresAt: string };
Every option has a default. The ones worth knowing at the start are spokenLanguage (leave it out and the language is detected), translate (a list of target languages), speakerLabels and latencyMode; the rest come up as they are needed below.
Hand the streamUrl to whatever captures the audio; from here on, no key is involved.
Connect to streamUrl as it is, with no header. While the session starts, the server sends session.loading heartbeats. Then comes session.ready. Send no audio before it. Audio sent early is not kept for later; the server treats it as an error, sends an error frame and closes the socket.
Once the socket is open, every failure takes the shape of that frame: {"type":"error","code":"…","message":"…","retryable":true|false}. The code is a closed list (invalid_audio, invalid_options, invalid_token, capacity_exceeded, session_expired, slow_consumer, engine_unavailable, internal), message is for your logs, and retryable tells you whether a fresh session is worth trying. Handle it from the first frame on:
const ws = new WebSocket(streamUrl); // the token is in the URL: no header
ws.binaryType = 'arraybuffer';
ws.addEventListener('message', (m: MessageEvent) => {
if (typeof m.data !== 'string') return;
const f = JSON.parse(m.data);
switch (f.type) {
case 'session.loading':
console.error('starting…');
break;
case 'session.ready':
startMicrophone(); // send audio only after session.ready
break;
case 'error':
console.error(`${f.code}: ${f.message}`);
if (f.retryable) scheduleReconnect(); // a new session, with backoff; see "Stop"
break;
}
});
A raw client has to handle two more things itself; the SDKs already do. First, a refused upgrade carries the API's usual HTTP error envelope: 401 invalid_token, 410 session_expired, or 503 unavailable with a Retry-After header to honour.
Second, saturation looks like silence, not an error. If no frame at all arrives within 30 seconds, treat it as capacity_exceeded and connect again; a new session usually gets through. If loading frames keep arriving but session.ready never does, treat it as engine_unavailable, which is also retryable. In the SDKs, RealtimeSession.connect resolves at session.ready and fails after readyDeadlineMs with the right code.
Once session.ready arrives, the server expects audio at the pace it is spoken.
Audio goes in as binary frames of raw PCM: 32-bit float, little-endian, 16 kHz, mono (pcm_f32le, the session's default audio format). Send it as it is captured, in chunks of about 100 to 250 ms, without waiting for the transcript between chunks.
The byte math is worth doing once. 16,000 samples a second at 4 bytes each is 64,000 bytes a second, so 100 ms is 6,400 bytes and 250 ms is 16,000. Speech models are commonly trained on 16 kHz mono, which is why so many realtime APIs ask for it, while microphones and browsers capture at 44.1 or 48 kHz and often in stereo. Something has to resample the audio and fold the channels into one before the bytes go out: here that is ffmpeg, and in the browser it is the SDK's microphone helper. Sending 48 kHz audio and declaring 16 kHz does not fail; it transcribes slow-motion speech, badly.
In Node, ffmpeg is the simplest capture device:
import { spawn } from 'node:child_process';
function startMicrophone(): void {
// Raw PCM: f32le, 16 kHz, mono, as binary frames. macOS: -f avfoundation -i :0, Linux: -f pulse -i default.
const mic = spawn('ffmpeg', ['-loglevel', 'error', '-f', 'avfoundation', '-i', ':0', '-f', 'f32le', '-ac', '1', '-ar', '16000', '-']);
mic.stdout.on('data', (chunk: Buffer) => {
// Backpressure: bytes the socket has accepted but not yet put on the wire.
// Two seconds of audio (128,000 bytes) queued means the link, not the server, is behind.
if (ws.bufferedAmount > 128_000) console.error(`link behind by ${ws.bufferedAmount} bytes`);
ws.send(new Uint8Array(chunk));
});
mic.on('close', () => ws.send(JSON.stringify({ type: 'eof' })));
process.once('SIGINT', () => mic.kill());
}
ws.send never blocks; it queues. bufferedAmount is the size of that queue, and on a healthy link it hovers near zero. Live audio cannot be paused, so the useful response to a growing queue is to notice it, not to drop frames. The server applies backpressure too. It accepts audio a few seconds ahead of real time (limits.audioBufferS), then stops reading until it has caught up. On the way back, commits are queued without loss while drafts are coalesced, so a slow reader sees fewer, fresher drafts. A client that cannot keep up even with the commits receives an error frame with slow_consumer and retryable: false, and is disconnected; if you see it, your message handler is doing too much work.
In a browser, streamMicrophone from @vocalstack/sdk/mic asks for the microphone, resamples to 16 kHz, and sends the chunks along with clock frames, which let the server mark gaps when the browser throttles the tab. The page gets the session from your route and never sees the key:
import { RealtimeSession } from '@vocalstack/sdk';
import { streamMicrophone } from '@vocalstack/sdk/mic';
export async function transcribeMicrophone(
stream: { sessionId: string; streamUrl: string },
render: (committed: string, tail: string) => void,
): Promise<void> {
const session = await RealtimeSession.connect(stream);
const mic = await streamMicrophone(session); // asks for the microphone, sends 16 kHz PCM
document.querySelector('#stop')?.addEventListener('click', () => {
void mic.stop();
session.eof();
});
let committed = '';
for await (const ev of session) {
if (ev.type === 'transcript.draft') render(committed, ev.segments.map((s) => s.text).join(' '));
if (ev.type === 'transcript.committed') {
committed += `${ev.segment.text} `;
render(committed, '');
}
}
}
That render(committed, tail) signature is the whole UI pattern; the next section explains why.
Two frame types build the transcript, and mixing them up is the most common mistake.
First, why there are two at all. A speech model decides what a word was partly from the words that follow it: "I scream" and "ice cream" sound the same until the sentence goes on. So a realtime recogniser emits a guess early, then revises it as more audio arrives, and only settles once it has heard enough. Every streaming speech API exposes this somehow. Some call the guesses "interim" or "partial" results and mark the settled ones is_final; the vocabulary varies, the discipline does not. Here the guess is a draft and the settled text is a commit.
transcript.draft is the live tail: the words heard since the last commit, still changing. Each draft is a full replacement of the previous one. Redraw the tail with its segments; never append it. Drafts carry no seq by design, so do not try to match them to the commits they become.
transcript.committed is final: one segment with seq (0, 1, 2, gapless), start, end, text, speaker and words with per-word timings. Append it and drop the draft it covers; the next draft starts where this commit ended.
committed: [seq 0] [seq 1]
draft: "how are" ← frame 1
draft: "how are you do" ← frame 2 REPLACES frame 1
committed: [seq 0] [seq 1] [seq 2 "How are you doing?"]
← the commit consumes the tail; drop the draft
A UI that appends drafts shows duplicated, flickering text. Your caption view develops a stutter the speaker never had. Render all committed segments, then the latest draft after them, and clear the draft when a commit arrives. A commit may contain words that never appeared in any draft.
Here are a few seconds of speech on the wire, from the fetch example above (translate: ["es"], sentence assembly off):
{"type":"session.loading"}
{"type":"session.ready"}
{"type":"transcript.draft","segments":[{"start":0.0,"text":"Good","speaker":0,"tentative":true}]}
{"type":"transcript.draft","segments":[{"start":0.0,"text":"Good morning every","speaker":0,"tentative":true}]}
{"type":"transcript.committed","segment":{"seq":0,"start":0.0,"end":1.4,"text":"Good morning, everyone.","speaker":0,"words":[{"start":0.0,"end":0.3,"text":"Good"},{"start":0.3,"end":0.8,"text":"morning,"},{"start":0.8,"end":1.4,"text":"everyone."}]}}
{"type":"transcript.translated","seq":0,"translations":{"es":"Buenos días a todos."}}
{"type":"transcript.draft","segments":[{"start":1.6,"text":"Thanks for","speaker":0,"tentative":true}]}
Two more frames sit around these. transcript.amended corrects a committed segment in place (same seq, new text or speaker) when a word in it has been corrected; apply it to the line you already drew. transcript.words streams word timings for the tail just before it commits. It needs events.words, which is on by default; a commit always embeds its words, whatever the event options say. Finally, ignore frame types you do not know: new ones are added without notice.
Here is the Python SDK's reading loop, with the terminal as the caption view. Sentence assembly is turned off here too, so translations arrive as transcript.translated, and the error frame surfaces as a VocalStackError raised out of the loop:
import asyncio
import os
from vocalstack import RealtimeSession, VocalStack, VocalStackError
MIC = ["ffmpeg", "-loglevel", "error", "-f", "avfoundation", "-i", ":0",
"-f", "f32le", "-ac", "1", "-ar", "16000", "-"]
async def main() -> None:
async with VocalStack("{{api}}", api_key=os.environ["VOCALSTACK_API_KEY"]) as vs:
created = await vs.realtime.create_session(
spoken_language="en", translate=["es"], sentences={"enabled": False}
)
session = await RealtimeSession.connect(created.session_id, created.stream_url)
mic = await asyncio.create_subprocess_exec(*MIC, stdout=asyncio.subprocess.PIPE)
async def feed() -> None:
assert mic.stdout is not None
while chunk := await mic.stdout.read(8000): # 125 ms of audio
await session.send_audio(chunk)
await session.eof() # the server flushes, then sends session.done
feeder = asyncio.create_task(feed())
try:
async with session:
async for event in session:
if event.type == "transcript.draft":
# The live tail: replace it, never append it.
tail = " ".join(s.text for s in event.segments)
print(f"\r\x1b[K\x1b[2m{tail}\x1b[0m", end="", flush=True)
elif event.type == "transcript.committed":
# Final: append it (seq 0, 1, 2...), and the tail it covers is gone.
print(f"\r\x1b[K{event.segment.text}", flush=True)
elif event.type == "transcript.translated":
print(f" es: {event.translations.get('es')}")
await feeder
print(f"done ({session.done_reason})", session.stats)
except VocalStackError as e:
# An `error` frame mid-stream, or a refused connect.
print(f"\nerror {e.code}: {e.message} (retryable: {e.retryable})")
finally:
if mic.returncode is None:
mic.kill()
asyncio.run(main())
Per-segment translation, as above, is the simple form. The form readers prefer gets a section of its own.
translate: ["es", "fr"] asks for translations as the session goes. How they arrive depends on sentence assembly (sentences: { enabled: true }), which is on by default.
With sentences off, each commit is followed a moment later by a transcript.translated frame naming its seq and carrying translations keyed by language; attach them to that segment. That is what the examples so far have used. It is simple and it reads like a commit log: one segment, one translation, the sentence boundary wherever the speaker drew breath.
With sentences on, the server assembles commits into whole sentences and translates those, which reads far better. Each transcript.sentence frame is a full replacement of one sentence, keyed by sentenceId. Its status goes from open to closed (or dropped, which means discard it), and its translations are provisional while it is open and final once it closes. Upsert by sentenceId, never append. The TypeScript SDK ships applySentence for exactly this:
import { applySentence, type SentenceSegment } from '@vocalstack/sdk';
let sentences: SentenceSegment[] = [];
for await (const ev of session) {
if (ev.type !== 'transcript.sentence') continue;
sentences = applySentence(sentences, ev.sentence); // upsert by sentenceId
if (ev.sentence.status === 'closed') console.log(`${ev.sentence.text}\n es: ${ev.sentence.translations?.es}`);
}
Translations follow the transcript by a second or two. If you want them spoken as well, add voice: ["es"] (it needs sentences). The Spanish for each closed sentence then arrives as a transcript.voice frame, carrying the clip as base64 Ogg/Opus at 24 kHz mono. A long sentence comes in pieces (part of parts) to play back to back. Provisional text is never spoken. The live translation guide covers voices in detail, and where the seconds go explains the delay chain.
Target languages are not fixed at creation.
A session.update text frame changes spokenLanguage, translate, speakerLabels, latencyMode, llmFix, voice or voiceByLang from that point on; fields you leave out are unchanged.
{"type":"session.update","translate":["es","fr"],"latencyMode":"fast"}
The SDKs expose it as session.update(…). translate replaces the whole list, and [] turns translation off. Segments already delivered are not re-transcribed or re-translated, and if you turn speaker labels on, speakers are labelled from that point only. What has been said stays said; the settings only govern what comes next.
When the audio ends, send {"type":"eof"} (session.eof() in the SDKs). The server transcribes what it still holds, sends the last commits, then session.done and closes. Sending cancel instead stops at once and drops the rest.
{"type":"session.done","reason":"eof","stats":{"audioS":30.0,"processingS":30.6,"realtimeFactor":0.98}}
reason is eof, max_duration (the session hit the plan's maximum length and ran the same flush) or cancelled. stats is measured by the server: audioS is the audio you sent, processingS is the wall-clock time the session ran, and realtimeFactor is their ratio. A live session lands just under 1, because the wall-clock time includes the final flush after your eof. A number well below that means the session took noticeably longer than the audio lasted; the first place to look is how fast the audio arrived. For a file, the same number says how many times faster than playback the transcript came back.
A session lasts at most limits.maxDurationS from GET /v1/realtime/capabilities, and its audio spends your credit per second, with no rounding, at your plan's live rate (pricing). The same socket also takes a whole file: create the session with input: { kind: "file" }, send the file's own bytes (mp3, wav, flac, ogg, m4a) and eof, and the transcript comes back many times faster than the audio plays, with transcript.progress frames along the way.
There is no resume. A dropped connection ends the session; reconnecting means creating a new one and streaming again from your live source, with seq back at 0. The reconnect itself is the standard pattern for any socket API, and worth getting right once:
retryable: true on an error frame, a 503 with its Retry-After, or 30 seconds without a frame, which stands for capacity_exceeded. invalid_token and session_expired mean a new streamUrl from your server, not another attempt with the old one.Keep the session-creating route cheap to call and treat a close as a fresh start. Having no resume is the price of storing nothing; if the transcript must outlive the socket, keep it.
When you want the same stream stored as a recording, corrected afterwards, exported, and read by an audience in their own languages, publish it to a live transcription instead. The go live guide walks the same PCM socket through a transcription and a broadcast link, and the listen guide shows the subscriber side, where every listener picks a language and a late joiner catches up from a lastSeq cursor. What that looks like for a room is the subject of one link, every language.
For a file you want kept, the stored route is transcribe a file with the API. The full frame reference, every field included, is the Stream a microphone guide in the API reference.