Stream live transcription over a WebSocket
Authenticate, stream 16 kHz PCM audio in short frames, and read the final text shortly after speech stops. A WebSocket guide to live speech to text, with code.
The WebSocket takes JSON text frames. The first frame must authenticate. The server detects the end of speech by loudness and replies with final text.
Message flow
address: wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1
> {"type":"auth","key":"YOUR_KEY"} < {"type":"ready"}
> {"type":"listen.start","language":"en","rate":16000,"interim":false} # optional: endpoint_ms (700), vad_threshold (400)
> {"type":"audio","seq":0,"data":"<base64 PCM16 mono, 100-500 ms>"} # seq counts up from 0
< {"type":"final","text":"...","seconds":4,"latency_ms":830}
> {"type":"listen.end"} # flushNode 22+ (global WebSocket), server side only
const ws = new WebSocket("wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1");
let seq = 0;
ws.onopen = () => ws.send(JSON.stringify({ type: "auth", key: process.env.SPEAKVORA_KEY }));
ws.onmessage = (m) => {
const msg = JSON.parse(m.data);
if (msg.type === "ready") {
ws.send(JSON.stringify({ type: "listen.start", language: "en", rate: 16000, interim: false }));
// send each 100-500 ms chunk of PCM16 mono as base64:
// ws.send(JSON.stringify({ type: "audio", seq: seq++, data: chunkBase64 }));
}
if (msg.type === "final") console.log(msg.text, msg.latency_ms + " ms");
};Tuning
- End-of-speech detection is loudness-based: raise
vad_thresholdin noisy rooms. - With
interim: trueyou also get partial results about every two seconds of speech; they re-transcribe the buffer and are billed as audio seconds. - The final text arrived about 0.8 seconds after speech stopped in our earlier test; the speed page has the method. It is a measurement, not a guarantee.
- Keep the key on your server and stream from there; never put it in a public web page.
Frame sizes and audio format
- Format: raw PCM, 16-bit signed little-endian, mono, base64 encoded in the
datafield. Allowed sample rates: 8000, 16000, 24000 and 48000 Hz; anything else getsinvalid_request. - Chunk size: 100 to 500 ms of audio per message. At 16 kHz that is 3,200 to 16,000 bytes (32 bytes per millisecond). The hard limit is 96,000 bytes per chunk.
- Sequence numbers:
seqstarts at 0, counts up by one and must not repeat. - Utterance length: an utterance is cut at
max_ms(default 25,000, never more than 28,000 ms) even if the speaker keeps going. - Flush: send
listen.endwhen your audio ends so the last words are transcribed instead of waiting for silence.
Settings you can send with listen.start
| Setting | Default | What it does |
|---|---|---|
| endpoint_ms | 700 | Silence, in milliseconds, after which the speech so far is final |
| vad_threshold | 400 | Loudness a chunk needs to count as speech; raise it in noisy rooms, lower it for quiet voices |
| interim_ms | 2000 | With interim on, how often partial results arrive |
| max_ms | 25000 | Longest single utterance (capped at 28,000) |
| min_voice_ms / drop_silence_ms | 200 / 4000 | Other defaults; the listening reply lists the values in effect |
Python, a WAV file sent as 200 ms chunks (pip install websockets)
import asyncio, base64, json, os, wave
import websockets
async def transcribe(path):
async with websockets.connect("wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1") as ws:
await ws.send(json.dumps({"type": "auth", "key": os.environ["SPEAKVORA_KEY"]}))
assert json.loads(await ws.recv())["type"] == "ready"
await ws.send(json.dumps({"type": "listen.start", "language": "en", "rate": 16000}))
with wave.open(path) as w: # 16 kHz, mono, 16-bit
seq = 0
while chunk := w.readframes(3200): # 3,200 frames = 200 ms
await ws.send(json.dumps({"type": "audio", "seq": seq, "data": base64.b64encode(chunk).decode()}))
seq += 1
await ws.send(json.dumps({"type": "listen.end"}))
while True:
msg = json.loads(await asyncio.wait_for(ws.recv(), 15))
if msg["type"] == "final":
return msg["text"]
if msg["type"] == "error":
raise RuntimeError(msg["code"] + ": " + msg["message"])
print(asyncio.run(transcribe("sample.wav")))Messages you will receive
| type | When | Useful fields |
|---|---|---|
| ready | After a valid auth | scope |
| listening | After listen.start | rate, interim and the settings in effect |
| partial | Interim results, if you turned them on | text, seconds |
| final | End of an utterance | text, seconds, latency_ms, from_seq, to_seq |
| error | Anything wrong | code, message, fix |
| pong | Reply to a ping message | t |
Errors and what to do
| code | Meaning | Fix |
|---|---|---|
| unauthorized | Key missing, wrong or revoked | Send auth as the first message with a current key |
| not_listening | Audio arrived before listen.start | Send listen.start first |
| invalid_request | Bad rate, empty chunk, chunk over 96,000 bytes or a missing seq | Check the format and frame size above |
| free_allowance_used | The free speech-to-text minutes for this month are used (they renew on the 1st, UTC) | Add a card or credit in the portal |
| upstream_unavailable | Transcription is briefly unavailable | Retry in a few seconds |
| unknown_type | A message type the server does not know | Use auth, listen.start, audio, listen.end, speak or ping |
What it costs
Speech to text is $0.0015 per minute of audio, with 60 free minutes per account every month (renewing on the 1st, UTC). Each transcription is counted in whole seconds, rounded up, with a one-second minimum. Interim partial results re-transcribe the buffer, so they are billed as audio seconds too; leave interim off unless you show live captions. See billing explained and the price comparison.
If no text comes back
- Did you send
listen.startbefore the first audio message? - Is the audio loud enough? Chunks below
vad_thresholdare treated as silence; try a lower value on a quiet microphone. - Is the sample rate in
listen.startthe real rate of your audio? A mismatch sounds like chipmunks or slow speech and transcribes badly. - Did you send
listen.endat the end? Without it the server waits for silence.