Omni realtime runtime

Omni is the realtime runtime behind PyAI Agents, exposed through one WebSocket. Speech, reasoning, tools, and turn-taking run in one loop at human conversational pace (about 390 ms median voice-to-voice, in-region; early measurement, not an SLA). You describe the agent in a configure frame. You do not assemble STT, an LLM, and TTS.

wss://api.pyai.com/v1/omni

Migrating an older Omni client? The former chat URL was discontinued on August 13, 2026. Follow the Omni endpoint migration guide for the exact URL, auth, query, configure, framing, and error-handling changes.

Send audio in. Receive spoken audio and events back. Omni handles speech recognition, reasoning, speech generation, voice activity detection, turn detection, and interruption handling inside the session.

Same runtime for UI and API agents

Create a saved agent in the console or with POST /v1/agents, then pass its returned agent_id as session_label when opening /v1/omni. The saved voice, instructions, knowledge and tools are loaded for the call. You can also configure sessions inline without saving a profile. The endpoint and omni:session scope stay the same.

One endpoint instead of a voice cascade

A traditional voice agent makes you assemble and operate this:

microphone → VAD → speech-to-text → LLM → text-to-speech → speaker

That diagram leaves out the hard parts: deciding when a turn is complete, stopping speech when the caller interrupts, carrying context between components, retrying failures, and keeping each hop responsive. With Omni, your application connects to one realtime endpoint. You describe the agent in a configure frame, then stream audio:

microphone → Omni → speaker
                 ↘ tools and knowledge

You still control the persona, voice, language, greeting, tools, and knowledge sources. PyAI runs the realtime conversation loop. Managed Agent profiles can also store an opt-in vocabulary list for up to five distinctive terms. The list is fixed at session start. It does not inherit organization Hear vocabulary, and inline Omni sessions do not accept a vocabulary override in this release. See Create agents via API. English agents can select Natural en1 (canonical id stock_aria_en). The configured acknowledgement reports the canonical voice id and voice_tier: "natural" while that tier is active. Omni Hindi uses the Standard voices hi1–hi4; there is no Hindi Natural tier (the former hi5–hi8 aliases are retired).

Connect

There is no agent resource you must create before opening a session. Authorize the WebSocket with an API key, then send the agent configuration:

import WebSocket from "ws";

function controlFrame(body: object) {
  return Buffer.concat([
    Buffer.from([0x03]),
    Buffer.from(JSON.stringify(body), "utf8"),
  ]);
}

const ws = new WebSocket(
  "wss://api.pyai.com/v1/omni?format=pcm16&rate=24000",
  ["pyai.v1", `pyai-key.${process.env.PYAI_API_KEY}`],
);

ws.on("open", () => {
  ws.send(controlFrame({
    type: "configure",
    voice_id: "stock_dorit_en_us",
    persona: "You are a helpful appointment scheduler.",
    tools: ["datetime"],
  }));
});

ws.on("message", (data, isBinary) => {
  if (!isBinary) return;
  const frame = Buffer.from(data as Buffer);
  if (frame[0] !== 0x03) return;
  const event = JSON.parse(frame.subarray(1).toString("utf8"));
  if (event.event === "configured") {
    console.log(event);
    // { "event": "configured", "voice_id": "stock_dorit_en_us", "tools": 1 }
  }
});

Success is a configured control frame within about two seconds. tools is the accepted count. After that, send PCM16 as 0x01 || audio_bytes. Server audio uses 0x01; plain UTF-8 caller-transcript deltas use 0x02; control JSON uses 0x03. Every server JSON body is keyed on event; the transcript body is not JSON. See the browser tutorial for capture, playback, and frame handling.

Choose how to build

Choose your path

API primitives, console Agent, or a framework adapter.

Launch an Agent

Create, test, and put a Call Now button on your site without a token broker.

Create agents via API

Store a profile, bind knowledge and tools, open Omni with session_label.

Build the browser client

Capture the microphone, mint a short-lived token, handle frames, and play audio.

Put Omni on a phone call

Bridge Twilio media, barge-in, DTMF, and call-control events.

Read the wire protocol

Binary audio framing, lifecycle events, tools, and session controls.

When to use a focused API instead

Use Hear when you only need transcripts or live captions. Use Speak when you already have text and only need audio. If you want to operate each component yourself, combine Hear with your own reasoning and speech output where Hear’s published sync/streaming language set fits the call.

Get started

Agents

Speech To Text (Hear)

Text To Speech (Speak)

Clone

Cast

Dub

Calls & intelligence

Frameworks

Integrations

Use cases

Reference

Same runtime for UI and API agents

One endpoint instead of a voice cascade

Connect

Choose how to build

Choose your path

Launch an Agent

Create agents via API

Build the browser client

Put Omni on a phone call

Read the wire protocol

When to use a focused API instead

​Same runtime for UI and API agents

​One endpoint instead of a voice cascade

​Connect

​Choose how to build

Choose your path

Launch an Agent

Create agents via API

Build the browser client

Put Omni on a phone call

Read the wire protocol

​When to use a focused API instead

Same runtime for UI and API agents

One endpoint instead of a voice cascade

Connect

Choose how to build

When to use a focused API instead