Omni wire protocol (v2)

The realtime WebSocket protocol for Omni, the all-in-one voice agent model: a hybrid speech-to-speech engine with a fused LLM brain that hears, reasons, calls your tools, grounds answers in your knowledge base, and speaks back in emotion-aware voices, all over this one socket. Omni runs on a single engine tier, so there is one protocol and one surface for agentic voice.

Endpoint: wss://api.pyai.com/v1/omni
Scope: omni:session (or the omni:* wildcard)
Status: GA

Migrating an older Omni client? The former chat URL was discontinued on August 13, 2026. Use the Omni endpoint migration guide before applying this canonical protocol reference.

Some browser-serving revisions also send a 0x03 advisory with exactly {"event":"transcript","role":"assistant","text":"…","final":true}. This describes the text submitted for speech synthesis; it does not confirm that the complete reply was played. Caller transcripts still use 0x02. For voice-quality evaluation, compare this advisory with a transcription of the captured audio. A machine-readable AsyncAPI 3.0 definition of this protocol ships alongside the OpenAPI contract at contracts/omni-asyncapi.yaml.

Field stability. Connect params, auth, the configure frame, the kb_endpoint callback, close codes, and metering are stable. Server→client lifecycle event payloads below document the confirmed envelope; individual fields marked provisional may gain keys. Branch only on event and ignore unknown fields. The official SDKs track these for you.

Frames are binary and type-prefixed, not text frames. Every server → client message is a binary frame whose first byte is a type tag: 0x01 = agent audio (PCM16), 0x02 = caller transcript (plain UTF-8 text), 0x03 = control/lifecycle (JSON keyed on event). A client that treats all binary as audio and parses only text frames for events will play control frames as a glitch and never see hello / session_started / transcript. Demux on the first byte (the official SDKs do this for you):

const buf = Buffer.from(data), t = buf[0], body = buf.subarray(1);
if (t === 0x01) playAudio(body);
else if (t === 0x02) onTranscriptDelta(body.toString("utf8"));
else if (t === 0x03) onEvent(JSON.parse(body.toString()));

1. Connect

wss://api.pyai.com/v1/omni?session_label=<label>&format=pcm16&rate=24000

Query param	Required	Values	Notes
`format`	no	`pcm16`	Audio sample format, both directions. Default `pcm16`. Load-bearing, the SDK sets it.
`rate`	no	`24000` · `16000` · `8000`	Caller-input sample rate. `24000` browser/WebRTC; `16000` wideband input; `8000` telephony. Default `24000`. Agent output is 24 kHz for both 24 kHz and 16 kHz input sessions; 8 kHz sessions receive 8 kHz output. Read `hello.audio_out`, it is authoritative.
`session_label`	no	any opaque tag (≤256 chars, header-safe)	Optional correlation tag. The value is echoed to your own `kb_endpoint`; when it matches an optional managed Agent profile, that profile is loaded. Any other value remains only a tag. Omit it if you need neither behavior. A malformed value (control chars / too long) is rejected with `400 invalid_session_label`.

Omni is zero-state by default, there is nothing to create first. The session is authorized by your key’s organization; the agent’s behavior can travel in the first configure frame below. A managed Agent profile is an optional convenience.

2. Auth

Browsers can’t set Authorization on a WebSocket upgrade, so pass the key as a subprotocol (browser-safe):

Sec-WebSocket-Protocol: pyai.v1, pyai-key.pyai_live_...

Server-side clients may instead append ?api_key=pyai_live_... to the URL. The key is validated at the edge and swapped for the internal engine credential, the customer key never reaches the engine. Don’t put the key in any other query param.

3. Audio frames

Send microphone audio as binary WebSocket messages in the negotiated format/rate (PCM16 little-endian), each prefixed with the 0x01 type tag. Receive the agent’s speech the same way, 0x01-prefixed binary frames you strip and play out as they arrive at the sample rate declared by hello.audio_out. Send caller frames continuously; the engine handles turn detection and barge-in server-side. For browser clients with a turn-0 consent_line or greeting, “continuously” means keeping the audio clock alive with digital-silence frames while that protected opening plays. Do not uplink microphone or speaker/self-audio until the playback graph is ready and the opening queue has drained. Then restore real mic samples; on later replies, preserve normal barge-in after a short echo-cancellation warm-up. Browser AEC alone is not an opening-audio gate.

const frame = new Uint8Array(pcm.byteLength + 1);
frame[0] = 0x01;                          // caller audio
frame.set(new Uint8Array(pcm.buffer), 1);
ws.send(frame);

The 0x01 tag is mandatory and omitting it fails without a client error. The engine demuxes every client frame on its first byte; an untagged PCM16 frame is dropped and counted internally, but no error frame is sent. You get a clean handshake, zero transcripts, and an agent that idles into “are you still there” — the session looks healthy and is simply deaf. The official SDKs tag for you.

Turn finalization: no commit frame

Omni has no client commit, EOF, done, or end-of-turn frame. Server-side turn detection advances while caller-audio frames arrive. Keep sending real-time 0x01 frames, including PCM16 zeroes while the caller is quiet, until reply audio or the next turn boundary. Digital zeroes are valid silence. A fixed trailing burst is not a commit signal: if a late transcript delta lands near the end of that burst, stopping all frames can leave no later tick on which to close the turn. File/WAV probes should send 20 ms zero frames until reply audio (with a bounded timeout), not send 800 ms once and then go idle. {"type":"session_ending"} and the WebSocket close end the session; they do not finalize a caller turn or request a reply. {"type":"commit"} is a Hear streaming-STT control, not an Omni control.

Transcript body

Every live 0x02 frame contains a non-empty UTF-8 text delta for the current caller turn. It is not JSON:

Can you help me?

Coalesce successive deltas into one partial caller row. The live wire carries no role or finality bit; applications may finalize the row when agent audio or a later turn boundary arrives. Official SDKs normalize each delta to {event:"transcript", role:"user", text, final:false, mode:"delta"} and retain bounded direct-JSON support only for older bridges.

4. Configure frame

Omni accepts complete behavior per session. Immediately after the upgrade, the client sends one JSON configure control frame carrying the agent’s behavior for this call. Control frames are keyed on type and carried as 0x03 || utf8_json, not as bare WebSocket text.

{
  "type": "configure",
  "voice_id": "stock_dorit_en_us",
  "voice_instruct": "Speak at a natural conversational pace with short pauses.",
  "persona": "You are a calm, concise booking assistant for ...",
  "kb_endpoint": "https://example.com/omni/context",
  "kb_token": "sk_your_callback_secret"
}

Fields: live vs roadmap

Field	Status	Meaning
`voice_id`	live	Voice to speak with, a stock, cloned, or designed id from `GET /v1/voices`. Multilingual stock voices also accept permanent short aliases (`es1`–`es3`, `fr1`, `de1`, `hi1`–`hi4`). Omit for the language default. Invalid non-English ids also serve that default; `configured.voice_id` reports the canonical id actually served.
`voice_instruct`	live	Optional delivery direction for instruct-capable voice tiers, up to 200 characters. Stored Agents use PyAI’s natural conversational pace when this is omitted; inline configure wins for the session.
`persona`	live	System prompt / role + instructions for the brain.
`kb_endpoint`	live	Optional customer-hosted URL the engine calls per turn for grounding (see §5). Hosted knowledge bases (`POST /v1/knowledgebases`, then bind to the agent) are the default path. Use `kb_endpoint` when you already run retrieval. See Knowledge bases.
`kb_token`	live	Bearer the engine presents to your `kb_endpoint`.
`greeting`	live	First line spoken on connect (turn 0). Set on `POST /v1/agents` or inline here. See Agent greeting messages.
`language`	live, staged	Accepted values are `en` (default), `fr`, `es`, `de`, and `hi`; inline wins over the Agent profile. As of 2026-08-13, public serving is enabled for `en`, `fr`, `es`, and `hi`; `de` falls back to English. Inspect `configured.language_active` and `language_fallback`; if fallback is unacceptable, stop the session. See Language support.
`model_tier`	roadmap	Opaque quality/cost tier. No-op today.
`consent_line`	live	Spoken before recording when `recordings_enabled=true` (agent profile or inline).
`meta`	live	Opaque customer object (≤16 keys / ≤4 KB). Customer fields are echoed on the post-call record and extraction webhook; PyAI-reserved internal fields are removed. `meta.external_id` lifts to top-level `external_id`. See Post-call extraction.
`tools[]`	live	Function calling. Three supported transports: hosted catalog tools (by name), client-loop (`tool_call` / `tool_result` on this socket), and server tools registered with `POST /v1/tools` then bound to the agent. An inline `endpoint` or `webhook_url` on `configure.tools[]` is rejected with `{event:"error", code:"unsupported_tool_transport"}`; the configure is not applied. See Omni tools.

Send only live fields for behavior you expect today. The gateway forwards unknown keys verbatim, but the engine ignores roadmap fields until they ship, sending them is a no-op, not an error. For per-call context (e.g. a user’s chart/profile), use persona plus the kb_endpoint callback rather than a roadmap field.

configured.voice_instruct_supported reports whether the selected synthesis path accepts delivery direction. In the current engine, English Natural supports it; Standard voices, including Elise and Gwen, do not. A saved or accepted voice_instruct value alone does not establish support. A missing capability field on an older engine means unknown, not supported. If delivery direction is required, require this field to be true before starting the call. Support means the direction can be passed to synthesis, not that the resulting audio is guaranteed to match a subjective style. If English Natural synthesis falls back to Standard during a call, Omni sends

{ "event": "voice_capabilities", "voice_tier": "standard", "voice_instruct_supported": false, "reason": "synthesis_fallback" }

. Use this event to update the capability state from configured.

Voice ids ending in _en_in are Indian-English voices, not Hindi voices. They remain compatible with an English-active session; they do not make that session Hindi. Hindi uses the Standard tier; see the pricing page for current tier treatment.

Agent profiles (`POST /v1/agents`)

Store persona, role, voice delivery, greeting message, recording disclosure, and tools once; connect with session_label={agent_id} so the engine loads them from your stored agent profile (no need to repeat greeting in configure unless overriding).

{
  "name": "Front desk",
  "voice_id": "stock_dorit_en_us",
  "greeting": "Hi, thanks for calling Acme. How can I help?",
  "persona_system_prompt": "You are a warm receptionist.",
  "recordings_enabled": true,
  "consent_line": "This call may be recorded for quality assurance."
}

Connect:

wss://api.pyai.com/v1/omni?session_label=agent_7f3a0b12&format=pcm16&rate=24000

Playback order when recordings are on: consent_line → greeting (turn 0) → conversation. Full walkthrough: Agent greeting messages · REST: POST /v1/agents.

4a. Function calling (`tools[]`)

Omni supports function calling on the live engine. Declare tools in the configure frame:

{
  "type": "configure",
  "persona": "You are a support agent. Use tools to look up orders.",
  "tools": [{
    "name": "get_order_status",
    "description": "Look up a customer order",
    "parameters": {
      "type": "object",
      "properties": { "order_id": { "type": "string" } },
      "required": ["order_id"]
    }
  }]
}

Client-loop (default). Omit any URL on the tool. When the brain selects a tool, the engine emits:

{ "event": "tool_call", "call_id": "…", "name": "get_order_status", "arguments": { "order_id": "123" } }

A write that needs caller confirmation first emits { "event": "tool_confirmation_required", "name": "…", "reason": "…" } instead of running the tool. After the caller confirms, the next turn may emit tool_call (client-loop) or invoke a registered server/hosted tool. Run a client-loop function in your app and reply on the same WebSocket:

{ "type": "tool_result", "call_id": "…", "result": { "status": "shipped" } }

On errors, return { "type": "tool_result", "call_id": "…", "error": "…" }. Timeouts: tools are load-bearing (unlike kb_endpoint grounding). Default per-tool budget is ~5 s (up to ~15 s). Long-running calls may trigger a brief spoken filler while the engine waits. Results over ~6 KB are truncated. Server tools: register a webhook with POST /v1/tools and bind it to the agent (PUT /v1/agents/{id}/tools). Do not put endpoint or webhook_url on the configure frame, that is rejected with { "event": "error", "code": "unsupported_tool_transport" } and the configure is not applied. Hosted catalog tools are enabled by name. Full guide: Omni function calling.

4.1 Call-control frames

Call-control tools run in engine mode on phone calls. Omni decides when a tool should fire and emits a 0x03 control frame. The transport carrying the phone leg must perform the carrier action. Every frame is JSON keyed on event. call_id identifies the engine tool invocation; tool arguments are spread alongside it:

Event	Stable shape	Transport action
`transfer_to_human`	`{ "event": "transfer_to_human", "call_id": "…", "destination": "+15551230000" }`	Transfer the live leg. `destination` comes from trusted per-agent tool configuration, not model output.
`send_dtmf`	`{ "event": "send_dtmf", "call_id": "…", "digits": "123#" }`	Send the requested touch tones.
`play_hold`	`{ "event": "play_hold", "call_id": "…", "seconds": 20 }`	Start hold audio. `seconds` may be omitted; stop on the next agent-audio frame or the local timeout.
`collect`	`{ "event": "collect", "call_id": "…", "field": "account_id", "kind": "speech" }`	Usually no immediate action. Keep forwarding speech and DTMF so the collected value returns through the normal input path. `kind` may be `speech` or `dtmf` and may be omitted.
`end_call`	`{ "event": "end_call", "call_id": "…" }`	Hang up the phone leg. A diagnostic `reason` or `terminal_reason` may also be present; do not branch the hangup on its text.

On a self-hosted transport, enabling one of these tools does not perform the carrier operation by itself. Implement the corresponding event handler first. The Twilio and FreeSWITCH guides show complete switches. Browser and in-app sessions have no phone leg, so these events are unavailable.

5. `kb_endpoint` grounding callback

If you set kb_endpoint, the engine calls your endpoint once per user turn to fetch grounding facts. This call comes from PyAI’s engine, not the browser. Request (engine → your endpoint):

POST <kb_endpoint>
Authorization: Bearer <kb_token>
Content-Type: application/json

{ "session_label": "<the connect-URL session_label, if any>", "query": "<the user's turn>" }

Response (your endpoint → engine): return grounding facts for the turn. A ready-to-inject context string and/or structured passages both work; keep it small and fast. Budget: the call has a hard ~300 ms timeout and is fail-open, on timeout or any error the engine proceeds with empty facts and never blocks the turn. Treat it as best-effort augmentation; keep it well under budget.

There is no documented retry of the callback, design it as a single best-effort call. session_label is how you route per-call context (e.g. look up the caller’s chart/profile by the tag you connected with).

6. Session lifecycle events (server → client)

On connect you receive hello followed by session_started, then turn and flush events interleaved with audio and plain-text 0x02 transcript deltas, ending in session_end. Every server JSON body is keyed only on event; the transcript body is not JSON.

Event	Payload (confirmed envelope)	Notes
`hello`	`{ "event": "hello", ... }`	Connection accepted; advertises protocol and audio formats.
`session_started`	`{ "event": "session_started", ... }`	Session is live; includes audio capabilities. Additional fields provisional.
`configured`	`{ "event": "configured", "voice_id": "...", "language_active": "en", "language_fallback": false, "audio_out": "pcm16@24000", "tools": 0 }`	The effective configuration. Compare `voice_id`, served language, fallback status, and output format with your requirements before starting application work. `tools` is the accepted count; an inline tool URL is never counted (the frame is rejected instead).
Transcript delta	plain UTF-8 text in a `0x02` frame	Append to the current caller transcript; there is no JSON discriminator or finality flag.
`turn`	`{ "event": "turn", "role": "user"\|"assistant" }`	Turn boundary.
`flush`	`{ "event": "flush" }`	User interrupted; stop local assistant playback immediately.
`tool_call`	`{ "event": "tool_call", "call_id": "...", "name": "...", "arguments": {} }`	Run a client tool and reply with a `type: tool_result` control frame.
`tool_confirmation_required`	`{ "event": "tool_confirmation_required", "name": "...", "reason": "..." }`	A write tool is waiting for the caller to confirm; it has not run.
`error`	`{ "event": "error", "code": "...", "message": "..." }`	Configure or session error. `unsupported_tool_transport` means an inline tool URL was sent; register via `POST /v1/tools` or use the client loop. Do not retry the same frame.
`transfer_to_human`	`{ "event": "transfer_to_human", "call_id": "...", "destination": "+15551230000" }`	Route the call to a human/PBX; see §4.1.
`send_dtmf`	`{ "event": "send_dtmf", "call_id": "...", "digits": "123#" }`	Send touch tones on the phone leg; see §4.1.
`play_hold`	`{ "event": "play_hold", "call_id": "...", ... }`	Start transport-owned hold media; see §4.1.
`collect`	`{ "event": "collect", "call_id": "...", ... }`	Continue forwarding caller speech/DTMF for collection; see §4.1.
`end_call`	`{ "event": "end_call", "call_id": "...", ... }`	Hang up the phone leg; see §4.1.
`session_end`	`{ "event": "session_end" }`	Session is closing; see the close code.

A flush with reason: "turn_merge" can include cancelled_turn, identifying a revoked generation when the caller resumes. Clear playback as usual. Preserve all turn_begin, audio and transcript events: a generation start is not proof of a spoken reply, and the cancellation receipt does not erase any received output.

7. Control frames (client → server)

Frame	Shape	Notes
`configure`	`{ "type": "configure", ... }`	Sent once, post-handshake (see §4).
`dtmf`	`{ "type": "dtmf", "digit": "5" }`	Forward a touch-tone digit into the session.
`context`	`{ "type": "context", "query": "...", "facts": ... }`	Optional client-push grounding for the current turn (the canonical path is the engine pulling from `kb_endpoint`).
`session_ending`	`{ "type": "session_ending" }`	Cleanly end the session. This is not a turn commit and does not request a reply.

8. Close codes

The server uses standard WebSocket close codes plus PyAI-specific application codes. Treat 4xxx-class application closes as non-retryable (fix the request); treat 1011-class closes as retryable with backoff.

Close code	Meaning	Retry?
`1000`	Normal closure	n/a
`4401`	Bad/expired key	No, fix credentials
`4403`	Missing/insufficient scope (need `omni:session`)	No
`4429`	Concurrency or rate cap hit	Yes, with backoff
`1011`	Transient engine/upstream error	Yes, with backoff

A malformed session_label is rejected before the upgrade as 400 invalid_session_label (an HTTP error, not a WS close).

9. Reconnect & retry

There is no mid-call session resume. A dropped socket means the session is over, reconnecting opens a new session and you must send a fresh configure frame. In-flight turn state is not preserved by PyAI.

Recommended pattern:

Retry on 1011 and 4429 with exponential backoff; do not retry 4401 / 4403 (fix the key/scope first).
Keep per-call state (the session_label, persona/context, a short running summary) in your backend so a reconnect can re-prime configure / kb_endpoint and continue gracefully.

10. Metering

Omni sessions report omni.minutes from session wall-clock duration. Check the pricing page for the current Omni and managed telephony rates, included usage, and billing rules. Realtime WebSocket sessions do not carry an x-pyai-units response header (that’s HTTP-only), reconcile realtime usage from your call records and usage data.

Authentication

Key handling and the WS subprotocol.

Errors & limits

Rate, concurrency, and the error catalog.

Telephony audio (8 kHz)

μ-law ↔ PCM16 at 8 kHz for phone legs.

Language support

What’s GA vs roadmap per language.

Get started

Agents

Speech To Text (Hear)

Text To Speech (Speak)

Clone

Cast

Dub

Calls & intelligence

Frameworks

Integrations

Use cases

Reference

1. Connect

2. Auth

3. Audio frames

Turn finalization: no commit frame

Transcript body

4. Configure frame

Fields: live vs roadmap

Agent profiles (`POST /v1/agents`)

4a. Function calling (`tools[]`)

4.1 Call-control frames

5. `kb_endpoint` grounding callback

6. Session lifecycle events (server → client)

7. Control frames (client → server)

8. Close codes

9. Reconnect & retry

10. Metering

See also

Authentication

Errors & limits

Telephony audio (8 kHz)

Language support

​1. Connect

​2. Auth

​3. Audio frames

​Turn finalization: no commit frame

​Transcript body

​4. Configure frame

​Fields: live vs roadmap

​Agent profiles (POST /v1/agents)

​4a. Function calling (tools[])

​4.1 Call-control frames

​5. kb_endpoint grounding callback

​6. Session lifecycle events (server → client)

​7. Control frames (client → server)

​8. Close codes

​9. Reconnect & retry

​10. Metering

​See also

Authentication

Errors & limits

Telephony audio (8 kHz)

Language support

1. Connect

2. Auth

3. Audio frames

Turn finalization: no commit frame

Transcript body

4. Configure frame

Fields: live vs roadmap

Agent profiles (`POST /v1/agents`)

4a. Function calling (`tools[]`)

4.1 Call-control frames

5. `kb_endpoint` grounding callback

6. Session lifecycle events (server → client)

7. Control frames (client → server)

8. Close codes

9. Reconnect & retry

10. Metering

See also