Stream live transcription over a WebSocket

Authenticate, stream 16 kHz PCM audio in short frames, and read the final text shortly after speech stops. A WebSocket guide to live speech to text, with code.

The WebSocket takes JSON text frames. The first frame must authenticate. The server detects the end of speech by loudness and replies with final text.

Message flow

address: wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1

> {"type":"auth","key":"YOUR_KEY"}                                    < {"type":"ready"}
> {"type":"listen.start","language":"en","rate":16000,"interim":false}   # optional: endpoint_ms (700), vad_threshold (400)
> {"type":"audio","seq":0,"data":"<base64 PCM16 mono, 100-500 ms>"}      # seq counts up from 0
< {"type":"final","text":"...","seconds":4,"latency_ms":830}
> {"type":"listen.end"}                                               # flush

Node 22+ (global WebSocket), server side only

const ws = new WebSocket("wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1");
let seq = 0;
ws.onopen = () => ws.send(JSON.stringify({ type: "auth", key: process.env.SPEAKVORA_KEY }));
ws.onmessage = (m) => {
  const msg = JSON.parse(m.data);
  if (msg.type === "ready") {
    ws.send(JSON.stringify({ type: "listen.start", language: "en", rate: 16000, interim: false }));
    // send each 100-500 ms chunk of PCM16 mono as base64:
    // ws.send(JSON.stringify({ type: "audio", seq: seq++, data: chunkBase64 }));
  }
  if (msg.type === "final") console.log(msg.text, msg.latency_ms + " ms");
};

Tuning

  • End-of-speech detection is loudness-based: raise vad_threshold in noisy rooms.
  • With interim: true you also get partial results about every two seconds of speech; they re-transcribe the buffer and are billed as audio seconds.
  • The final text arrived about 0.8 seconds after speech stopped in our earlier test; the speed page has the method. It is a measurement, not a guarantee.
  • Keep the key on your server and stream from there; never put it in a public web page.

Frame sizes and audio format

  • Format: raw PCM, 16-bit signed little-endian, mono, base64 encoded in the data field. Allowed sample rates: 8000, 16000, 24000 and 48000 Hz; anything else gets invalid_request.
  • Chunk size: 100 to 500 ms of audio per message. At 16 kHz that is 3,200 to 16,000 bytes (32 bytes per millisecond). The hard limit is 96,000 bytes per chunk.
  • Sequence numbers: seq starts at 0, counts up by one and must not repeat.
  • Utterance length: an utterance is cut at max_ms (default 25,000, never more than 28,000 ms) even if the speaker keeps going.
  • Flush: send listen.end when your audio ends so the last words are transcribed instead of waiting for silence.

Settings you can send with listen.start

SettingDefaultWhat it does
endpoint_ms700Silence, in milliseconds, after which the speech so far is final
vad_threshold400Loudness a chunk needs to count as speech; raise it in noisy rooms, lower it for quiet voices
interim_ms2000With interim on, how often partial results arrive
max_ms25000Longest single utterance (capped at 28,000)
min_voice_ms / drop_silence_ms200 / 4000Other defaults; the listening reply lists the values in effect

Python, a WAV file sent as 200 ms chunks (pip install websockets)

import asyncio, base64, json, os, wave
import websockets

async def transcribe(path):
    async with websockets.connect("wss://48f8pzwtee.execute-api.us-east-1.amazonaws.com/v1") as ws:
        await ws.send(json.dumps({"type": "auth", "key": os.environ["SPEAKVORA_KEY"]}))
        assert json.loads(await ws.recv())["type"] == "ready"
        await ws.send(json.dumps({"type": "listen.start", "language": "en", "rate": 16000}))
        with wave.open(path) as w:                      # 16 kHz, mono, 16-bit
            seq = 0
            while chunk := w.readframes(3200):           # 3,200 frames = 200 ms
                await ws.send(json.dumps({"type": "audio", "seq": seq, "data": base64.b64encode(chunk).decode()}))
                seq += 1
        await ws.send(json.dumps({"type": "listen.end"}))
        while True:
            msg = json.loads(await asyncio.wait_for(ws.recv(), 15))
            if msg["type"] == "final":
                return msg["text"]
            if msg["type"] == "error":
                raise RuntimeError(msg["code"] + ": " + msg["message"])

print(asyncio.run(transcribe("sample.wav")))

Messages you will receive

typeWhenUseful fields
readyAfter a valid authscope
listeningAfter listen.startrate, interim and the settings in effect
partialInterim results, if you turned them ontext, seconds
finalEnd of an utterancetext, seconds, latency_ms, from_seq, to_seq
errorAnything wrongcode, message, fix
pongReply to a ping messaget

Errors and what to do

codeMeaningFix
unauthorizedKey missing, wrong or revokedSend auth as the first message with a current key
not_listeningAudio arrived before listen.startSend listen.start first
invalid_requestBad rate, empty chunk, chunk over 96,000 bytes or a missing seqCheck the format and frame size above
free_allowance_usedThe free speech-to-text minutes for this month are used (they renew on the 1st, UTC)Add a card or credit in the portal
upstream_unavailableTranscription is briefly unavailableRetry in a few seconds
unknown_typeA message type the server does not knowUse auth, listen.start, audio, listen.end, speak or ping

What it costs

Speech to text is $0.0015 per minute of audio, with 60 free minutes per account every month (renewing on the 1st, UTC). Each transcription is counted in whole seconds, rounded up, with a one-second minimum. Interim partial results re-transcribe the buffer, so they are billed as audio seconds too; leave interim off unless you show live captions. See billing explained and the price comparison.

If no text comes back

  • Did you send listen.start before the first audio message?
  • Is the audio loud enough? Chunks below vad_threshold are treated as silence; try a lower value on a quiet microphone.
  • Is the sample rate in listen.start the real rate of your audio? A mismatch sounds like chipmunks or slow speech and transcribes badly.
  • Did you send listen.end at the end? Without it the server waits for silence.