> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Omni wire protocol (v2)
> Full reference for the Omni realtime WebSocket: connect URL, auth, audio and configure frames, kb_endpoint grounding, lifecycle, close codes, metering.
The realtime WebSocket protocol for **Omni**, the all-in-one voice agent model: a
hybrid speech-to-speech engine with a fused LLM brain that hears, reasons, calls
your tools, grounds answers in your knowledge base, and speaks back in
emotion-aware voices, all over this one socket. Omni runs on a single engine
tier, so there is one protocol and one surface for agentic voice.
* **Native endpoint:** `wss://api.pyai.com/v1/omni`
* **OpenAI-realtime-compatible alias:** `wss://api.pyai.com/v1/realtime?model=pyai-omni-realtime`
* **Deprecated alias:** `wss://api.pyai.com/v2/omni/chat` (works during the deprecation window; new integrations MUST use `/v1/omni`)
* **Scope:** `omni:session` (or the `omni:*` wildcard)
* **Status:** GA
A machine-readable **AsyncAPI 3.0** definition of this protocol ships alongside
the OpenAPI contract at `contracts/omni-asyncapi.yaml`.
**Field stability.** Connect params, auth, the `configure` frame, the
`kb_endpoint` callback, close codes, and metering are **stable**. Server→client
**lifecycle event payloads** below document the confirmed envelope; individual
fields marked *provisional* may gain keys, branch on the event `type`/`event`
name and ignore unknown fields. The official SDKs track these for you.
**Frames are binary and type-prefixed, not text frames.** Every server → client
message is a binary frame whose **first byte is a type tag**: `0x01` = agent
**audio** (PCM16), `0x02` = **transcript** (JSON), `0x03` = **control/lifecycle**
(JSON keyed on `event`). A client that treats all binary as audio and parses only
*text* frames for events will play control frames as a glitch and never see
`hello` / `session_started` / `transcript`. **Demux on the first byte** (the
official SDKs do this for you):
```js theme={null}
const buf = Buffer.from(data), t = buf[0], body = buf.subarray(1);
if (t === 0x01) playAudio(body);
else if (t === 0x02) onTranscript(JSON.parse(body.toString()));
else if (t === 0x03) onEvent(JSON.parse(body.toString()));
```
## 1. Connect
```
wss://api.pyai.com/v1/omni?session_label=&format=pcm16&rate=24000
```
| Query param | Required | Values | Notes |
| --------------- | -------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `format` | no | `pcm16` | Audio sample format, both directions. Default `pcm16`. **Load-bearing**, the SDK sets it. |
| `rate` | no | `24000` · `16000` · `8000` | Sample rate (Hz). `24000` browser/WebRTC; `16000` wideband telephony; `8000` for an 8 kHz G.711/Twilio leg (μ-law companding only, **no resampling**, see [Telephony audio](/reference/telephony-audio)). Default `24000`. **Load-bearing**, sent on the connect URL; the gateway preserves it verbatim. |
| `session_label` | no | any opaque tag (≤256 chars, header-safe) | Optional correlation tag. PyAI stores **no** per-session state; the value is echoed to your own `kb_endpoint` so you can branch per session. Omit it if you don't need correlation. A malformed value (control chars / too long) is rejected with `400 invalid_agent_id`. |
| `agent_id` | no | any opaque tag | **Deprecated alias** for `session_label`. If both are sent, `session_label` wins. (`agent` is the legacy alias on the deprecated `/v2/omni/chat` URL.) |
Omni is **zero-state, there is nothing to create first.** The session is
authorized by your key's **organization**; the agent's behavior travels in the
first `configure` frame (below), not a pre-built record.
## 2. Auth
Browsers can't set `Authorization` on a WebSocket upgrade, so pass the key as a
subprotocol (browser-safe):
```
Sec-WebSocket-Protocol: pyai-key.pyai_live_...
```
Server-side clients may instead append `?api_key=pyai_live_...` to the URL. The
key is validated at the edge and swapped for the internal engine credential, the
customer key never reaches the engine. Don't put the key in any other query param.
## 3. Audio frames
Send microphone audio as **binary** WebSocket messages in the negotiated
`format`/`rate` (PCM16 little-endian), each prefixed with the **`0x01`** type
tag. Receive the agent's speech the same way, `0x01`-prefixed binary frames you
strip and play out as they arrive. Send frames continuously; the engine handles
turn detection and barge-in server-side.
```js theme={null}
const frame = new Uint8Array(pcm.byteLength + 1);
frame[0] = 0x01; // caller audio
frame.set(new Uint8Array(pcm.buffer), 1);
ws.send(frame);
```
**The `0x01` tag is mandatory and omitting it fails silently.** The engine
demuxes every client frame on its first byte and has **no default branch**, so an
untagged PCM16 frame is dropped with no error, no log, and no counter. You get a
clean handshake, zero transcripts, and an agent that idles into "are you still
there" — the session looks healthy and is simply deaf. The official SDKs tag for
you.
## 4. Configure frame
Omni is **stateless** on PyAI. Immediately after the upgrade, the client sends one
JSON **`configure`** control frame carrying the agent's behavior for this call.
Control frames are keyed on **`type`**.
```json theme={null}
{
"type": "configure",
"voice_id": "stock_dorit_en_us",
"persona": "You are a calm, concise booking assistant for ...",
"kb_endpoint": "https://example.com/omni/context",
"kb_token": "sk_your_callback_secret"
}
```
### Fields: live vs roadmap
| Field | Status | Meaning |
| -------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `voice_id` | **live** | Voice to speak with, a stock, cloned, or designed id from `GET /v1/voices`. |
| `persona` | **live** | System prompt / role + instructions for the brain. |
| `kb_endpoint` | **live** | Your customer-hosted URL the engine calls per turn for grounding (see §5). PyAI does not host the KB. |
| `kb_token` | **live** | Bearer the engine presents to your `kb_endpoint`. |
| `greeting` | **live** | First line spoken on connect (turn 0). Set on [`POST /v1/agents`](/api-reference) or inline here. See [Agent greeting messages](/guides/agent-greeting). |
| `language` | **live** (staged rollout) | Session language, end to end: `en` (default), `fr`, `es`, `de`, `hi`. Also settable on the agent profile ([`POST /v1/agents`](/api-reference)); an inline value here wins for the session. Fail-safe: a language that is unknown or not yet enabled for your traffic falls back to `en` and the call proceeds, the `configured` ack reports `language_active` (what is actually being served) and `language_fallback: true`, and the call bills as what was served. Per-language availability is being enabled in stages, see [Language support](/reference/language-support). |
| `model_tier` | roadmap | Opaque quality/cost tier. **No-op today.** |
| `consent_line` | **live** | Spoken **before** recording when `recordings_enabled=true` (agent profile or inline). |
| `meta` | **live** | Opaque customer object (≤16 keys / ≤4 KB). Echoed on the post-call record and extraction webhook; `meta.external_id` lifts to top-level `external_id`. See [Post-call extraction](/guides/post-call-extraction). |
| `tools[]` | **live** | Functions the brain may call mid-call. **Client-loop (default):** omit `endpoint`, the engine emits `tool_call` on the WebSocket and your client runs it and replies with `tool_result`. **Engine-POST (optional):** set `endpoint` to an HTTPS URL the engine POSTs to. See [Omni tools](/guides/omni-tools). |
Send only **live** fields for behavior you expect today. The gateway forwards unknown keys verbatim,
but the **engine** ignores roadmap fields until they ship, sending them is a
no-op, not an error. For per-call context (e.g. a user's chart/profile), use
`persona` plus the `kb_endpoint` callback rather than a roadmap field.
### Agent profiles (`POST /v1/agents`)
Store persona, voice, **greeting message**, recording disclosure, and tools once;
connect with `session_label={agent_id}` so the engine loads them from your
stored agent profile (no need to repeat `greeting` in `configure` unless overriding).
```json theme={null}
{
"name": "Front desk",
"voice_id": "stock_dorit_en_us",
"greeting": "Hi, thanks for calling Acme. How can I help?",
"persona_system_prompt": "You are a warm receptionist.",
"recordings_enabled": true,
"consent_line": "This call may be recorded for quality assurance."
}
```
Connect:
```
wss://api.pyai.com/v1/omni?session_label=agent_7f3a0b12&format=pcm16&rate=24000
```
**Playback order when recordings are on:** `consent_line` → greeting (turn 0) → conversation.
Full walkthrough: [Agent greeting messages](/guides/agent-greeting) · REST: [`POST /v1/agents`](/api-reference).
## 4a. Function calling (`tools[]`)
Omni supports **function calling** on the live engine. Declare tools in the
`configure` frame:
```json theme={null}
{
"type": "configure",
"persona": "You are a support agent. Use tools to look up orders.",
"tools": [{
"name": "get_order_status",
"description": "Look up a customer order",
"parameters": {
"type": "object",
"properties": { "order_id": { "type": "string" } },
"required": ["order_id"]
}
}]
}
```
**Client-loop (default)**, no `endpoint` on the tool. When the brain selects a
tool, the engine emits:
```json theme={null}
{ "type": "tool_call", "call_id": "…", "name": "get_order_status", "arguments": { "order_id": "123" } }
```
Run the function in your app and reply on the same WebSocket:
```json theme={null}
{ "type": "tool_result", "call_id": "…", "result": { "status": "shipped" } }
```
On errors, return `{ "type": "tool_result", "call_id": "…", "error": "…" }`.
**Timeouts:** tools are **load-bearing** (unlike `kb_endpoint` grounding).
Default per-tool budget is **\~5 s** (up to \~15 s). Long-running calls may trigger
a brief spoken filler while the engine waits. Results over \~6 KB are truncated.
**Engine-POST (optional):** add `"endpoint": "https://…"` on a tool definition.
Register reusable webhook tools via [`GET /v1/tools`](/api-reference).
## 5. `kb_endpoint` grounding callback
If you set `kb_endpoint`, the engine calls **your** endpoint once per user turn to
fetch grounding facts. This call comes from PyAI's engine, not the browser.
**Request (engine → your endpoint):**
```
POST
Authorization: Bearer
Content-Type: application/json
{ "session_label": "", "query": "" }
```
**Response (your endpoint → engine):** return grounding facts for the turn. A
ready-to-inject `context` string and/or structured passages both work; keep it
small and fast.
**Budget:** the call has a **hard \~300 ms timeout and is fail-open**, on timeout
or any error the engine proceeds with **empty facts** and **never blocks the
turn**. Treat it as best-effort augmentation; keep it well under budget.
There is **no documented retry** of the callback, design it as a single
best-effort call. `session_label` is how you route per-call context (e.g. look up
the caller's chart/profile by the tag you connected with). The legacy `agent_id`
key may also appear in the request body for back-compat.
## 6. Session lifecycle events (server → client)
On connect you receive a greeting/hello frame followed by `session_started`,
then turn/transcript/barge-in events interleaved with binary audio, ending in
`session_end`.
| Event | Payload (confirmed envelope) | Notes |
| ------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `session_started` | `{ "type": "session_started", ... }` | Session is live; includes the resolved voice and audio caps (`format`/`rate`). *Additional fields provisional.* |
| `transcript` | `{ "type": "transcript", "role": "user"\|"assistant", "text": "...", "final": false }` | Incremental/final transcript. `role` may also arrive as `speaker`; finality may arrive as `final`, `is_final`, or `kind: "final"`. |
| `turn` | `{ "type": "turn", ... }` | Turn boundary (who is speaking). *Fields provisional.* |
| `barge_in` | `{ "type": "barge_in" }` | User interrupted; assistant audio is being cut. Stop playback immediately. Stable alias: `flush`. |
| `transfer_to_human` | `{ "type": "transfer_to_human", "to": "+15551230000" }` | Route the call to a human/PBX (e.g. `uuid_transfer` in the FreeSWITCH guide). |
| `session_end` | `{ "type": "session_end", ... }` | Session is closing; see the close code. Stable alias: `session_ending`. |
## 7. Control frames (client → server)
| Frame | Shape | Notes |
| ----------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `configure` | `{ "type": "configure", ... }` | Sent once, post-handshake (see §4). |
| `dtmf` | `{ "type": "dtmf", "digit": "5" }` | Forward a touch-tone digit into the session. |
| `context` | `{ "type": "context", "query": "...", "facts": ... }` | Optional client-push grounding for the current turn (the canonical path is the engine pulling from `kb_endpoint`). |
## 8. Close codes
The server uses standard WebSocket close codes plus PyAI-specific application
codes. Treat `4xxx`-class application closes as **non-retryable** (fix the
request); treat `1011`-class closes as **retryable with backoff**.
| Close code | Meaning | Retry? |
| ---------- | ------------------------------------------------ | ------------------- |
| `1000` | Normal closure | n/a |
| `4401` | Bad/expired key | No, fix credentials |
| `4403` | Missing/insufficient scope (need `omni:session`) | No |
| `4429` | Concurrency or rate cap hit | Yes, with backoff |
| `1011` | Transient engine/upstream error | Yes, with backoff |
A malformed `session_label`/`agent_id` is rejected **before** the upgrade as
`400 invalid_agent_id` (an HTTP error, not a WS close).
## 9. Reconnect & retry
There is **no mid-call session resume.** A dropped socket means the session is
over, reconnecting opens a **new** session and you must send a fresh `configure`
frame. In-flight turn state is not preserved by PyAI.
Recommended pattern:
* Retry on `1011` and `4429` with exponential backoff; do **not** retry `4401` /
`4403` (fix the key/scope first).
* Keep per-call state (the `session_label`, persona/context, a short running
summary) in **your** backend so a reconnect can re-prime `configure` /
`kb_endpoint` and continue gracefully.
## 10. Metering
Omni sessions meter as **`omni.minutes`** by session wall-clock duration, **billed
per second** (no minimum; an empty/failed call bills nothing). The billed quantity
is duration-minutes (fractional) × the [Omni list rate](/pricing-and-metering).
Realtime WebSocket sessions do **not** carry an `x-pyai-units` response header
(that's HTTP-only), reconcile realtime usage from your call records and usage
data.
## 11. Migration from `/v2/omni/chat`
1. Change the URL from `/v2/omni/chat` to `/v1/omni`.
2. Rename the `agent` query param to `session_label` (the `agent_id`/`agent`
aliases still work).
3. Keep the same `pyai-key.` subprotocol auth, unchanged.
4. The bridge-only HTTP surfaces (`/v2/omni/health`, `/v2/omni/calls…`) are
retired; there is no `/v1/omni` HTTP equivalent.
## See also
Key handling and the WS subprotocol.
Rate, concurrency, and the error catalog.
μ-law ↔ PCM16 at 8 kHz for phone legs.
What's GA vs roadmap per language.