Omni wire protocol (v2)

The realtime WebSocket protocol for Omni, the all-in-one voice agent model: a hybrid speech-to-speech engine with a fused LLM brain that hears, reasons, calls your tools, grounds answers in your knowledge base, and speaks back in emotion-aware voices, all over this one socket. Omni runs on a single engine tier, so there is one protocol and one surface for agentic voice.

Native endpoint: wss://api.pyai.com/v1/omni
OpenAI-realtime-compatible alias: wss://api.pyai.com/v1/realtime?model=pyai-omni-realtime
Deprecated alias: wss://api.pyai.com/v2/omni/chat (works during the deprecation window; new integrations MUST use /v1/omni)
Scope: omni:session (or the omni:* wildcard)
Status: GA

A machine-readable AsyncAPI 3.0 definition of this protocol ships alongside the OpenAPI contract at contracts/omni-asyncapi.yaml.

Field stability. Connect params, auth, the configure frame, the kb_endpoint callback, close codes, and metering are stable. Server→client lifecycle event payloads below document the confirmed envelope; individual fields marked provisional may gain keys, branch on the event type/event name and ignore unknown fields. The official SDKs track these for you.

Frames are binary and type-prefixed, not text frames. Every server → client message is a binary frame whose first byte is a type tag: 0x01 = agent audio (PCM16), 0x02 = transcript (JSON), 0x03 = control/lifecycle (JSON keyed on event). A client that treats all binary as audio and parses only text frames for events will play control frames as a glitch and never see hello / session_started / transcript. Demux on the first byte (the official SDKs do this for you):

const buf = Buffer.from(data), t = buf[0], body = buf.subarray(1);
if (t === 0x01) playAudio(body);
else if (t === 0x02) onTranscript(JSON.parse(body.toString()));
else if (t === 0x03) onEvent(JSON.parse(body.toString()));

1. Connect

wss://api.pyai.com/v1/omni?session_label=<label>&format=pcm16&rate=24000

Query param	Required	Values	Notes
`format`	no	`pcm16`	Audio sample format, both directions. Default `pcm16`. Load-bearing, the SDK sets it.
`rate`	no	`24000` · `16000` · `8000`	Sample rate (Hz). `24000` browser/WebRTC; `16000` wideband telephony; `8000` for an 8 kHz G.711/Twilio leg (μ-law companding only, no resampling, see Telephony audio). Default `24000`. Load-bearing, sent on the connect URL; the gateway preserves it verbatim.
`session_label`	no	any opaque tag (≤256 chars, header-safe)	Optional correlation tag. PyAI stores no per-session state; the value is echoed to your own `kb_endpoint` so you can branch per session. Omit it if you don’t need correlation. A malformed value (control chars / too long) is rejected with `400 invalid_agent_id`.
`agent_id`	no	any opaque tag	Deprecated alias for `session_label`. If both are sent, `session_label` wins. (`agent` is the legacy alias on the deprecated `/v2/omni/chat` URL.)

Omni is zero-state, there is nothing to create first. The session is authorized by your key’s organization; the agent’s behavior travels in the first configure frame (below), not a pre-built record.

2. Auth

Browsers can’t set Authorization on a WebSocket upgrade, so pass the key as a subprotocol (browser-safe):

Sec-WebSocket-Protocol: pyai-key.pyai_live_...

Server-side clients may instead append ?api_key=pyai_live_... to the URL. The key is validated at the edge and swapped for the internal engine credential, the customer key never reaches the engine. Don’t put the key in any other query param.

3. Audio frames

Send microphone audio as binary WebSocket messages in the negotiated format/rate (PCM16 little-endian), each prefixed with the 0x01 type tag. Receive the agent’s speech the same way, 0x01-prefixed binary frames you strip and play out as they arrive. Send frames continuously; the engine handles turn detection and barge-in server-side.

const frame = new Uint8Array(pcm.byteLength + 1);
frame[0] = 0x01;                          // caller audio
frame.set(new Uint8Array(pcm.buffer), 1);
ws.send(frame);

The 0x01 tag is mandatory and omitting it fails silently. The engine demuxes every client frame on its first byte and has no default branch, so an untagged PCM16 frame is dropped with no error, no log, and no counter. You get a clean handshake, zero transcripts, and an agent that idles into “are you still there” — the session looks healthy and is simply deaf. The official SDKs tag for you.

4. Configure frame

Omni is stateless on PyAI. Immediately after the upgrade, the client sends one JSON configure control frame carrying the agent’s behavior for this call. Control frames are keyed on type.

{
  "type": "configure",
  "voice_id": "stock_dorit_en_us",
  "persona": "You are a calm, concise booking assistant for ...",
  "kb_endpoint": "https://example.com/omni/context",
  "kb_token": "sk_your_callback_secret"
}

Fields: live vs roadmap

Field	Status	Meaning
`voice_id`	live	Voice to speak with, a stock, cloned, or designed id from `GET /v1/voices`.
`persona`	live	System prompt / role + instructions for the brain.
`kb_endpoint`	live	Your customer-hosted URL the engine calls per turn for grounding (see §5). PyAI does not host the KB.
`kb_token`	live	Bearer the engine presents to your `kb_endpoint`.
`greeting`	live	First line spoken on connect (turn 0). Set on `POST /v1/agents` or inline here. See Agent greeting messages.
`language`	live (staged rollout)	Session language, end to end: `en` (default), `fr`, `es`, `de`, `hi`. Also settable on the agent profile (`POST /v1/agents`); an inline value here wins for the session. Fail-safe: a language that is unknown or not yet enabled for your traffic falls back to `en` and the call proceeds, the `configured` ack reports `language_active` (what is actually being served) and `language_fallback: true`, and the call bills as what was served. Per-language availability is being enabled in stages, see Language support.
`model_tier`	roadmap	Opaque quality/cost tier. No-op today.
`consent_line`	live	Spoken before recording when `recordings_enabled=true` (agent profile or inline).
`meta`	live	Opaque customer object (≤16 keys / ≤4 KB). Echoed on the post-call record and extraction webhook; `meta.external_id` lifts to top-level `external_id`. See Post-call extraction.
`tools[]`	live	Functions the brain may call mid-call. Client-loop (default): omit `endpoint`, the engine emits `tool_call` on the WebSocket and your client runs it and replies with `tool_result`. Engine-POST (optional): set `endpoint` to an HTTPS URL the engine POSTs to. See Omni tools.

Send only live fields for behavior you expect today. The gateway forwards unknown keys verbatim, but the engine ignores roadmap fields until they ship, sending them is a no-op, not an error. For per-call context (e.g. a user’s chart/profile), use persona plus the kb_endpoint callback rather than a roadmap field.

Agent profiles (`POST /v1/agents`)

Store persona, voice, greeting message, recording disclosure, and tools once; connect with session_label={agent_id} so the engine loads them from your stored agent profile (no need to repeat greeting in configure unless overriding).

{
  "name": "Front desk",
  "voice_id": "stock_dorit_en_us",
  "greeting": "Hi, thanks for calling Acme. How can I help?",
  "persona_system_prompt": "You are a warm receptionist.",
  "recordings_enabled": true,
  "consent_line": "This call may be recorded for quality assurance."
}

Connect:

wss://api.pyai.com/v1/omni?session_label=agent_7f3a0b12&format=pcm16&rate=24000

Playback order when recordings are on: consent_line → greeting (turn 0) → conversation. Full walkthrough: Agent greeting messages · REST: POST /v1/agents.

4a. Function calling (`tools[]`)

Omni supports function calling on the live engine. Declare tools in the configure frame:

{
  "type": "configure",
  "persona": "You are a support agent. Use tools to look up orders.",
  "tools": [{
    "name": "get_order_status",
    "description": "Look up a customer order",
    "parameters": {
      "type": "object",
      "properties": { "order_id": { "type": "string" } },
      "required": ["order_id"]
    }
  }]
}

Client-loop (default), no endpoint on the tool. When the brain selects a tool, the engine emits:

{ "type": "tool_call", "call_id": "…", "name": "get_order_status", "arguments": { "order_id": "123" } }

Run the function in your app and reply on the same WebSocket:

{ "type": "tool_result", "call_id": "…", "result": { "status": "shipped" } }

On errors, return { "type": "tool_result", "call_id": "…", "error": "…" }. Timeouts: tools are load-bearing (unlike kb_endpoint grounding). Default per-tool budget is ~5 s (up to ~15 s). Long-running calls may trigger a brief spoken filler while the engine waits. Results over ~6 KB are truncated. Engine-POST (optional): add "endpoint": "https://…" on a tool definition. Register reusable webhook tools via GET /v1/tools.

5. `kb_endpoint` grounding callback

If you set kb_endpoint, the engine calls your endpoint once per user turn to fetch grounding facts. This call comes from PyAI’s engine, not the browser. Request (engine → your endpoint):

POST <kb_endpoint>
Authorization: Bearer <kb_token>
Content-Type: application/json

{ "session_label": "<the connect-URL session_label, if any>", "query": "<the user's turn>" }

Response (your endpoint → engine): return grounding facts for the turn. A ready-to-inject context string and/or structured passages both work; keep it small and fast. Budget: the call has a hard ~300 ms timeout and is fail-open, on timeout or any error the engine proceeds with empty facts and never blocks the turn. Treat it as best-effort augmentation; keep it well under budget.

There is no documented retry of the callback, design it as a single best-effort call. session_label is how you route per-call context (e.g. look up the caller’s chart/profile by the tag you connected with). The legacy agent_id key may also appear in the request body for back-compat.

6. Session lifecycle events (server → client)

On connect you receive a greeting/hello frame followed by session_started, then turn/transcript/barge-in events interleaved with binary audio, ending in session_end.

Event	Payload (confirmed envelope)	Notes
`session_started`	`{ "type": "session_started", ... }`	Session is live; includes the resolved voice and audio caps (`format`/`rate`). Additional fields provisional.
`transcript`	`{ "type": "transcript", "role": "user"\|"assistant", "text": "...", "final": false }`	Incremental/final transcript. `role` may also arrive as `speaker`; finality may arrive as `final`, `is_final`, or `kind: "final"`.
`turn`	`{ "type": "turn", ... }`	Turn boundary (who is speaking). Fields provisional.
`barge_in`	`{ "type": "barge_in" }`	User interrupted; assistant audio is being cut. Stop playback immediately. Stable alias: `flush`.
`transfer_to_human`	`{ "type": "transfer_to_human", "to": "+15551230000" }`	Route the call to a human/PBX (e.g. `uuid_transfer` in the FreeSWITCH guide).
`session_end`	`{ "type": "session_end", ... }`	Session is closing; see the close code. Stable alias: `session_ending`.

7. Control frames (client → server)

Frame	Shape	Notes
`configure`	`{ "type": "configure", ... }`	Sent once, post-handshake (see §4).
`dtmf`	`{ "type": "dtmf", "digit": "5" }`	Forward a touch-tone digit into the session.
`context`	`{ "type": "context", "query": "...", "facts": ... }`	Optional client-push grounding for the current turn (the canonical path is the engine pulling from `kb_endpoint`).

8. Close codes

The server uses standard WebSocket close codes plus PyAI-specific application codes. Treat 4xxx-class application closes as non-retryable (fix the request); treat 1011-class closes as retryable with backoff.

Close code	Meaning	Retry?
`1000`	Normal closure	n/a
`4401`	Bad/expired key	No, fix credentials
`4403`	Missing/insufficient scope (need `omni:session`)	No
`4429`	Concurrency or rate cap hit	Yes, with backoff
`1011`	Transient engine/upstream error	Yes, with backoff

A malformed session_label/agent_id is rejected before the upgrade as 400 invalid_agent_id (an HTTP error, not a WS close).

9. Reconnect & retry

There is no mid-call session resume. A dropped socket means the session is over, reconnecting opens a new session and you must send a fresh configure frame. In-flight turn state is not preserved by PyAI.

Recommended pattern:

Retry on 1011 and 4429 with exponential backoff; do not retry 4401 / 4403 (fix the key/scope first).
Keep per-call state (the session_label, persona/context, a short running summary) in your backend so a reconnect can re-prime configure / kb_endpoint and continue gracefully.

10. Metering

Omni sessions meter as omni.minutes by session wall-clock duration, billed per second (no minimum; an empty/failed call bills nothing). The billed quantity is duration-minutes (fractional) × the Omni list rate. Realtime WebSocket sessions do not carry an x-pyai-units response header (that’s HTTP-only), reconcile realtime usage from your call records and usage data.

11. Migration from `/v2/omni/chat`

Change the URL from /v2/omni/chat to /v1/omni.
Rename the agent query param to session_label (the agent_id/agent aliases still work).
Keep the same pyai-key.<key> subprotocol auth, unchanged.
The bridge-only HTTP surfaces (/v2/omni/health, /v2/omni/calls…) are retired; there is no /v1/omni HTTP equivalent.

Authentication

Key handling and the WS subprotocol.

Errors & limits

Rate, concurrency, and the error catalog.

Telephony audio (8 kHz)

μ-law ↔ PCM16 at 8 kHz for phone legs.

Language support

What’s GA vs roadmap per language.

Get started

Guides

Integrations

Use cases

Reference

1. Connect

2. Auth

3. Audio frames

4. Configure frame

Fields: live vs roadmap

Agent profiles (`POST /v1/agents`)

4a. Function calling (`tools[]`)

5. `kb_endpoint` grounding callback

6. Session lifecycle events (server → client)

7. Control frames (client → server)

8. Close codes

9. Reconnect & retry

10. Metering

11. Migration from `/v2/omni/chat`

See also

Authentication

Errors & limits

Telephony audio (8 kHz)

Language support

​1. Connect

​2. Auth

​3. Audio frames

​4. Configure frame

​Fields: live vs roadmap

​Agent profiles (POST /v1/agents)

​4a. Function calling (tools[])

​5. kb_endpoint grounding callback

​6. Session lifecycle events (server → client)

​7. Control frames (client → server)

​8. Close codes

​9. Reconnect & retry

​10. Metering

​11. Migration from /v2/omni/chat

​See also

Authentication

Errors & limits

Telephony audio (8 kHz)

Language support

1. Connect

2. Auth

3. Audio frames

4. Configure frame

Fields: live vs roadmap

Agent profiles (`POST /v1/agents`)

4a. Function calling (`tools[]`)

5. `kb_endpoint` grounding callback

6. Session lifecycle events (server → client)

7. Control frames (client → server)

8. Close codes

9. Reconnect & retry

10. Metering

11. Migration from `/v2/omni/chat`

See also