> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Build a browser voice agent

> Capture the mic, stream PCM16 to Omni over WebSocket, and play the agent's voice back, a full talking agent in the browser in about 10 minutes.

By the end of this guide you'll have a single HTML page that opens your mic,
streams audio to an [Omni](/realtime/omni-protocol) agent in real time, plays the
agent's reply through your speakers, and renders the live transcript as you both
talk. No framework, no build step, just the Web Audio API and a WebSocket.

<Note>
  **Run the complete example.** Scaffold this guide's full, CI-tested code in one
  command, no clone:

  ```bash theme={null}
  npm create pyai-app@latest omni-browser-widget
  ```

  Or browse it: [`omni-browser-widget`](https://github.com/atomsai/pyai-examples/tree/main/omni-browser-widget).
</Note>

## How it fits together

```mermaid theme={null}
flowchart LR
  mic[getUserMedia] --> cap[AudioWorklet capture]
  cap -->|Float32 to PCM16| up[24 kHz PCM16 frames]
  up -->|binary WS| omni[Omni wss/v1/omni]
  omni -->|binary WS: agent audio| play[AudioWorklet playback]
  omni -->|text WS: events + transcript| ui[Transcript UI]
  play --> spk[Speakers]
```

One binary envelope shares the socket: byte `0x01` carries PCM16 audio, `0x02`
carries UTF-8 JSON transcript payloads, and `0x03` carries UTF-8 JSON control
payloads. Caller audio must also include the leading `0x01`; outbound control
frames use `0x03` plus JSON keyed by `type`. See the wire protocol before
implementing a transport.

<Info>
  This guide is correct on transport, codecs, resampling, and the event *behaviors*
  you build against. The exact JSON payloads for each event are defined by the
  [Omni wire protocol](/realtime/omni-protocol), which is the source of truth, we
  isolate event handling behind a single function so you can fill in field names
  once you've checked the reference.
</Info>

## Prerequisites

<Steps>
  <Step title="A key, that's it">
    Omni is **zero-state**: there's nothing to create first. The session is
    authorized by your key's org, and the agent's behavior travels in a
    `configure` frame after connect. Grab a `pyai_test_` sandbox key to build
    against, it works instantly with hard daily caps and no billing.
  </Step>

  <Step title="A local static server">
    `getUserMedia` requires a secure context, which includes `http://localhost`.
    Any static server works: `npx serve`, `python3 -m http.server`, etc.
  </Step>
</Steps>

<Warning>
  A `pyai_test_` key is fine for a local experiment. **Never ship any long-lived
  key or publishable token in a production voice agent.** Use the hosted website
  widget from Agent → Connect, or have your backend call
  `POST /v1/omni/sessions` and hand the browser only the returned short-lived,
  origin-locked session token.
</Warning>

## Pick one sample rate and stick to it

The single biggest source of "it sounds like chipmunks/robots" bugs is a sample
rate mismatch. Omni speaks **PCM16 little-endian**; for browser/WebRTC use
**24 kHz**, declared on the URL as `?format=pcm16&rate=24000`.

The clean trick: ask the browser for a 24 kHz `AudioContext` so the mic capture,
the worklets, and the wire all agree, **no resampling needed**.

```js theme={null}
const ctx = new AudioContext({ sampleRate: 24000 });
```

Most browsers honor this. If yours pins the context to its hardware rate
(commonly 48 kHz), you have two options: keep the 24 kHz context and let the
worklet see 24 kHz directly, or run at 48 kHz and **decimate 2:1** on capture
(take every other sample) / **upsample 2:1** on playback (duplicate each sample).
The math is exact because 48000 / 24000 = 2. We use the 24 kHz context path; the
fallback is one `if` away and called out in the code.

## Build it

<Steps>
  <Step title="Capture worklet: mic Float32 → PCM16">
    An `AudioWorkletProcessor` runs on the audio thread and hands you 128-sample
    blocks. Buffer them into \~20 ms frames (480 samples @ 24 kHz), convert
    Float32 `[-1, 1]` to little-endian Int16, and post the bytes to the main
    thread.

    ```js capture-processor.js theme={null}
    // 20 ms @ 24 kHz = 480 samples per frame.
    const FRAME = 480;

    class CaptureProcessor extends AudioWorkletProcessor {
      constructor() {
        super();
        this._buf = new Int16Array(FRAME);
        this._n = 0;
      }

      process(inputs) {
        const ch = inputs[0]?.[0];
        if (!ch) return true;
        for (let i = 0; i < ch.length; i++) {
          // clamp then scale: negatives use 0x8000, positives 0x7FFF
          const s = Math.max(-1, Math.min(1, ch[i]));
          this._buf[this._n++] = s < 0 ? s * 0x8000 : s * 0x7fff;
          if (this._n === FRAME) {
            // transfer the ArrayBuffer to avoid a copy
            const out = this._buf.slice();
            this.port.postMessage(out.buffer, [out.buffer]);
            this._n = 0;
          }
        }
        return true;
      }
    }

    registerProcessor("capture-processor", CaptureProcessor);
    ```
  </Step>

  <Step title="Playback worklet: a ring buffer of agent audio">
    Agent audio arrives in bursts; speakers consume it at a steady 24 kHz. Bridge
    the two with a ring buffer. Crucially, expose a `clear` message, that's how
    barge-in stays snappy (next step).

    ```js playback-processor.js theme={null}
    const CAP = 24000 * 10; // up to 10 s of buffered audio

    class PlaybackProcessor extends AudioWorkletProcessor {
      constructor() {
        super();
        this._ring = new Float32Array(CAP);
        this._r = 0;
        this._w = 0;
        this.port.onmessage = (e) => {
          if (e.data === "clear") {
            this._r = this._w = 0; // drop everything queued (barge-in)
            return;
          }
          const pcm = new Int16Array(e.data);
          for (let i = 0; i < pcm.length; i++) {
            this._ring[this._w] = pcm[i] / 0x8000; // Int16 → Float32
            this._w = (this._w + 1) % CAP;
          }
        };
      }

      process(_inputs, outputs) {
        const out = outputs[0][0];
        for (let i = 0; i < out.length; i++) {
          out[i] = this._r === this._w ? 0 : this._ring[this._r];
          if (this._r !== this._w) this._r = (this._r + 1) % CAP;
        }
        return true;
      }
    }

    registerProcessor("playback-processor", PlaybackProcessor);
    ```
  </Step>

  <Step title="Connect to Omni and wire the audio loop">
    Open the WebSocket with the key as a subprotocol (browser-safe, browsers
    can't set headers on the upgrade), forward capture frames as binary, and feed
    binary replies into the playback ring. The official `@pyai/sdk` ships
    `realtimeURL` / `realtimeSubprotocol` helpers so you don't hand-build the URL.

    <CodeGroup>
      ```js Vanilla JS theme={null}
      const API_KEY = "pyai_test_..."; // publishable/short-lived token in prod

      // Omni is zero-state, no agent to create. Optionally add
      // &session_label=<tag> to brand the session in your kb_endpoint.
      const url = "wss://api.pyai.com/v1/omni?format=pcm16&rate=24000";

      async function start() {
        const ctx = new AudioContext({ sampleRate: 24000 });
        await ctx.audioWorklet.addModule("capture-processor.js");
        await ctx.audioWorklet.addModule("playback-processor.js");

        const stream = await navigator.mediaDevices.getUserMedia({
          audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
        });

        const src = ctx.createMediaStreamSource(stream);
        const capture = new AudioWorkletNode(ctx, "capture-processor");
        const playback = new AudioWorkletNode(ctx, "playback-processor");
        src.connect(capture);          // mic → capture (no audible monitor)
        playback.connect(ctx.destination); // playback → speakers

        const ws = new WebSocket(url, [`pyai-key.${API_KEY}`]);
        ws.binaryType = "arraybuffer";

        // set the agent's voice + behavior on connect (zero-state configure)
        ws.onopen = () => ws.send(JSON.stringify({
          type: "configure",
          voice_id: "stock_dorit_en_us",
          persona: "You are a friendly receptionist.",
        }));

        // mic frames → server
        capture.port.onmessage = (e) => {
          if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
        };

        ws.onmessage = (e) => {
          if (typeof e.data === "string") {
            handleEvent(JSON.parse(e.data), playback); // see next step
          } else {
            playback.port.postMessage(e.data); // agent audio → speakers
          }
        };

        ws.onclose = (e) => console.log("closed", e.code, e.reason);
        window.stop = () => { ws.close(); stream.getTracks().forEach((t) => t.stop()); };
      }
      ```

      ```ts @pyai/sdk theme={null}
      import PyAI from "@pyai/sdk";

      const pyai = new PyAI({ apiKey: API_KEY });
      // omni.connect() handles the frame-key asymmetry for you: it sends a
      // type-keyed `configure` on open and parses event-keyed server frames into
      // callbacks, so you can't trip the "configured but zero turns" bug.
      const omni = pyai.omni.connect({
        rate: 24000, // zero-state, sessionLabel is optional, not required
        configure: { voice_id: "stock_dorit_en_us", persona: "You are a friendly receptionist." },
        onAudio: (chunk) => playback.port.postMessage(chunk), // agent audio → speakers
        onBargeIn: () => playback.port.postMessage("clear"),  // drop stale audio
        onTranscript: (f) => renderTranscript(f),
        onError: (e) => console.error(e),
      });
      // mic frames → server
      capture.port.onmessage = (e) => omni.sendAudio(e.data);
      ```
    </CodeGroup>

    <Warning>
      **Frame-key asymmetry, the #1 Omni bug.** Your **outbound** control frames
      (`configure`, `dtmf`) are keyed on **`type`**; the **inbound** server frames
      (`hello`, `session_started`, `configured`, `flush`, …) are keyed on
      **`event`**. The edge is transparent, so a mis-keyed `{"event":"configure"}`
      is **acked but silently dropped**, you get a connected session that never
      speaks, with no error. Send `{"type":"configure",…}`; parse inbound on
      `event`. The `@pyai/sdk` `omni.connect()` helper does both for you.
    </Warning>
  </Step>

  <Step title="Handle events: transcript + barge-in">
    Text frames are JSON session events. You build against the **known event
    names** below; render whatever transcript fields your agent emits. Keep all of
    this in one function so the payload details live in exactly one place.

    ```js theme={null}
    function handleEvent(msg, playback) {
      // ⚠️ Inbound server frames are keyed on `event`, NOT `type`. (Your
      // outbound control frames, e.g. configure, are keyed on `type`.) Switching
      // on `msg.type` here is the #1 Omni bug: every frame falls through.
      switch (msg.event) {
        case "hello":            // connection accepted, before session_started
        case "session_started":  // session is live; start talking
        case "configured":       // ack for your configure frame
          break;

        case "flush":            // (a.k.a. barge_in)
        case "barge_in":
          // Barge-in: you started speaking, so the agent's queued audio is
          // now stale. Drop it immediately for a snappy turn-handoff.
          playback.port.postMessage("clear");
          break;

        case "transfer_to_human": // agent decided to escalate
        case "session_end":       // server is closing the session
          break;

        default:
          // Transcript / partials arrive as JSON text frames too, render the
          // text fields your agent emits here (exact shape: protocol reference).
          renderTranscript(msg);
      }
    }
    ```

    <Info>
      The exact fields on each event (and the transcript payload shape) are defined
      in the [Omni wire protocol](/realtime/omni-protocol). Map them inside
      `renderTranscript` / the cases above once you've confirmed names there, the
      transport and event *names* used here are stable.
    </Info>
  </Step>
</Steps>

## Run it

Drop `capture-processor.js`, `playback-processor.js`, and an `index.html` (a
**Start** button calling `start()`) in one folder and serve it:

```bash theme={null}
npx serve .   # then open http://localhost:3000
```

Click **Start**, allow the mic, and say hello. You should hear the agent reply
within a few hundred milliseconds and see the transcript fill in. Talk over it, the agent's audio should cut out as soon as you speak (that's the `flush` →
`clear` path).

## Barge-in, latency & quality

* **Barge-in** is the difference between a demo and a product. The server detects
  your speech and sends `flush`; your only job is to **stop playing queued agent
  audio right then**, that's the single `playback.port.postMessage("clear")`
  call. Don't wait for the socket to drain.
* **Keep frames small (\~20 ms).** Smaller frames lower latency; much smaller and
  you pay per-message overhead. 480 samples @ 24 kHz is a good default.
* **Let the browser do AEC.** `echoCancellation: true` stops the agent's own
  voice from being captured and looping back as user speech.
* **Don't add your own jitter buffer on top of the worklet ring**, the ring is
  already the buffer. Extra queuing only adds latency.
* **Resume the AudioContext from a user gesture.** Browsers start it suspended;
  call `ctx.resume()` inside the click handler if playback is silent.

## Troubleshooting

| Symptom                                                                               | Likely cause                                                                                                  | Fix                                                                                                                                                                                                            |
| ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Connects and `configured` acks, but the agent never speaks (zero turns, no error)** | **`configure` keyed on `event` instead of `type`**, the engine drops the unknown key and runs with no persona | Send `{"type":"configure",…}` (not `"event"`). Parse inbound on `event`. The `@pyai/sdk` `omni.connect()` helper handles both sides.                                                                           |
| Agent sounds high/low-pitched or sped up                                              | Sample-rate mismatch                                                                                          | Ensure the `AudioContext`, worklets, and `?rate=` all agree on 24 kHz; if the context forced 48 kHz, decimate/upsample 2:1                                                                                     |
| Connection closes immediately                                                         | Bad credential, missing scope, or a malformed `session_label`                                                 | Check the close code against [Errors & limits](/errors-and-limits); `4401` = bad key, `4403` = missing `omni:session` scope, `400 invalid_agent_id` = malformed `session_label` (or the deprecated `agent_id`) |
| `403 origin_not_allowed` on close                                                     | Publishable token origin not allow-listed                                                                     | Add your origin in the console (production tokens only)                                                                                                                                                        |
| No mic prompt / `getUserMedia` throws                                                 | Insecure context                                                                                              | Serve over `http://localhost` or HTTPS, not `file://` or a LAN IP                                                                                                                                              |
| Agent hears itself / echoes                                                           | AEC off, or playback feeding back                                                                             | Set `echoCancellation: true`; never connect `capture` to `destination`                                                                                                                                         |
| Choppy or robotic playback                                                            | Ring buffer starved or overrun                                                                                | Confirm 20 ms frames; check the socket isn't backpressured                                                                                                                                                     |
| `429 concurrency_limit_exceeded`                                                      | Too many live sessions                                                                                        | Close old sockets; raise the limit on your plan                                                                                                                                                                |

## Make the agent speak first (greeting)

By default the **caller speaks first**, Omni listens, then replies. To open the
call with a greeting ("Hi, thanks for calling, how can I help?"), set the native
`greeting` field in your `configure` frame, it's **live**, and the engine speaks
it as turn 0. The **pre-roll pattern** below is an alternative for when you'd
rather render the opening audio client-side:

<Steps>
  <Step title="Synthesize the greeting once with Speak">
    Call `POST /v1/audio/speech` with `response_format: "pcm"` at your session
    rate and cache the bytes (it's the same line every call).
  </Step>

  <Step title="Play it the moment the session is live">
    On `session_started`, push the cached PCM into your playback ring **before**
    you start streaming mic audio. The agent appears to speak first; normal
    turn-taking + barge-in take over from there.
  </Step>
</Steps>

<Note>
  The native `greeting` field in the `configure` frame is **live**: set it and the
  engine speaks your opening line as turn 0 (one billing line, no extra hop), so the
  pre-roll above is optional. For agent profiles, set **Greeting message** in the
  [console Agents builder](/guides/agent-greeting) instead of sending `greeting` every
  connect. The connect URL's `session_label` is still just an opaque correlation tag
  when you are not using an agent profile.
</Note>

## Next steps

<CardGroup cols={2}>
  <Card title="Omni wire protocol" href="/realtime/omni-protocol">Exact event payloads, close codes, and golden frames.</Card>
  <Card title="Phone agent with Twilio" href="/guides/twilio-voice-agent">Bridge the same Omni session to a real phone number.</Card>
  <Card title="FreeSWITCH integration" href="/guides/freeswitch-voice-agent">Fork SIP/PSTN audio into Omni.</Card>
  <Card title="Errors & limits" href="/errors-and-limits">Close codes, rate limits, and concurrency.</Card>
</CardGroup>