> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Integrate Omni with FreeSWITCH

> Fork channel audio (L16/16 kHz) to an Omni agent over WebSocket, with barge-in via uuid_break, live transfer via uuid_transfer, and DTMF over ESL.

If your telephony already runs on FreeSWITCH, you can drop an
[Omni](/realtime/omni-protocol) agent onto any channel without leaving your
dialplan. FreeSWITCH forks the call's audio to a small WebSocket bridge, the
bridge relays it to Omni and plays the agent's reply back into the channel, and
you drive call control (barge-in, transfer, DTMF) over the Event Socket.

<Note>
  **Run the complete example.** Scaffold this guide's full bridge in one command, no clone:

  ```bash theme={null}
  npm create pyai-app@latest freeswitch-omni-voice-agent
  ```

  Or browse it: [`freeswitch-omni-voice-agent`](https://github.com/atomsai/pyai-examples/tree/main/freeswitch-omni-voice-agent).
</Note>

## How it fits together

```mermaid theme={null}
flowchart LR
  caller([SIP/PSTN caller]) --> fs[FreeSWITCH]
  fs <-->|mod_audio_stream fork: L16 16 kHz binary| bridge[Your Node bridge]
  bridge <-->|binary WS: PCM16 16 kHz| omni[Omni wss/v1/omni]
  bridge -.->|ESL: uuid_break / uuid_transfer| fs
  fs -.->|ESL: DTMF events| bridge
```

`mod_audio_stream` (or `mod_audio_fork`) forks channel audio as **L16, signed
linear 16-bit PCM, at 16 kHz**, and plays audio you send back into the same
channel. L16/16 kHz is byte-for-byte the same as Omni's **PCM16 at 16 kHz**, so
run Omni with `?rate=16000` and the audio path is a **straight passthrough, no
resampling**. Call control rides a separate **ESL** (Event Socket) connection.

<Info>
  This guide is correct on transport, the fork codec/rate, and the Omni event
  *behaviors*. The exact JSON payloads of Omni events (notably the `dtmf` frame you
  forward) are defined by the [Omni wire protocol](/realtime/omni-protocol), they
  live in one helper so updating them is a one-line change.
</Info>

## Prerequisites

<Steps>
  <Step title="FreeSWITCH with a fork module">
    `mod_audio_stream` (bidirectional; recommended) or `mod_audio_fork` loaded and
    in `modules.conf.xml`. Confirm with `fs_cli -x "module_exists mod_audio_stream"`.
  </Step>

  <Step title="Event Socket access">
    The inbound Event Socket enabled (default `127.0.0.1:8021`, password
    `ClueCon`). Lock this down to localhost or your bridge host.
  </Step>

  <Step title="A key">
    A `pyai_test_` key, supplied to the bridge as an environment variable. Omni is
    zero-state, no agent to create; optionally pick a `session_label` to tag each
    call in your `kb_endpoint`.
  </Step>
</Steps>

## Project layout

```text theme={null}
freeswitch-omni-bridge/
├── server.js               # /fork WebSocket bridge + ESL control
├── package.json
├── .env                    # PYAI_API_KEY, FS_HOST, FS_PASSWORD, ...
└── dialplan/omni-agent.xml # the extension that starts the fork
```

```json package.json theme={null}
{
  "name": "freeswitch-omni-bridge",
  "type": "module",
  "scripts": { "start": "node server.js" },
  "dependencies": {
    "@fastify/websocket": "^11.0.0",
    "dotenv": "^16.4.0",
    "fastify": "^5.0.0",
    "modesl": "^1.6.0",
    "ws": "^8.18.0"
  }
}
```

## Build it

<Steps>
  <Step title="Dialplan: fork the channel to your bridge">
    Answer the call, then start `mod_audio_stream` toward your bridge at **16 kHz
    mono**, and `park` the channel so it stays up while audio streams. Pass the
    channel UUID on the URL so the bridge can issue ESL commands for it.

    ```xml dialplan/omni-agent.xml theme={null}
    <include>
      <extension name="omni-agent">
        <condition field="destination_number" expression="^9000$">
          <action application="answer"/>
          <action application="audio_stream"
                  data="start wss://${bridge_host}/fork?uuid=${uuid} mono 16k"/>
          <action application="park"/>
        </condition>
      </extension>
    </include>
    ```

    <Tip>
      With `mod_audio_fork` the app is `uuid_audio_fork`/`audio_fork`; the arguments
      (mix type `mono`, rate `16k`) are the same. Keep the rate at `16k` so it lines
      up with Omni's `rate=16000` and no resampling is needed.
    </Tip>
  </Step>

  <Step title="Bridge: relay fork audio ↔ Omni">
    FreeSWITCH connects to `/fork` and streams binary L16 frames. Forward them to
    Omni untouched, and stream Omni's binary frames straight back into the channel.

    ```js server.js (audio bridge) theme={null}
    import "dotenv/config";
    import Fastify from "fastify";
    import websocket from "@fastify/websocket";
    import WebSocket from "ws";
    import esl from "modesl";

    const { PYAI_API_KEY, FS_HOST, FS_PASSWORD } = process.env;

    const app = Fastify();
    await app.register(websocket);

    const calls = new Map(); // uuid → { omni, fsWS }

    // Zero-state: no agent_id needed. Add &session_label=<tag> to brand the call.
    const omniURL = "wss://api.pyai.com/v1/omni?format=pcm16&rate=16000";

    app.get("/fork", { websocket: true }, (fsWS, req) => {
      const uuid = new URL(`http://x${req.url}`).searchParams.get("uuid");
      const omni = new WebSocket(omniURL, [`pyai-key.${PYAI_API_KEY}`]);
      calls.set(uuid, { omni, fsWS });

      // caller audio (L16 16 kHz) → Omni (PCM16 16 kHz): identical bytes
      fsWS.on("message", (data, isBinary) => {
        if (isBinary && omni.readyState === WebSocket.OPEN) omni.send(data);
      });

      // agent audio → back into the channel; text frames are session events
      omni.on("message", (data, isBinary) => {
        if (isBinary) fsWS.send(data);
        else handleOmniEvent(JSON.parse(data.toString()), uuid);
      });

      const cleanup = () => { calls.delete(uuid); omni.close(); };
      fsWS.on("close", cleanup);
      omni.on("close", () => fsWS.close());
    });
    ```

    <Warning>
      L16 and PCM16 are both signed 16-bit, but **endianness must match**. If you
      hear static or white noise, your fork is emitting big-endian, byte-swap each
      16-bit sample (`buf.swap16()`) before forwarding, and again on the way back.
    </Warning>
  </Step>

  <Step title="ESL: barge-in, transfer, and DTMF">
    One Event Socket connection drives every call. Subscribe to DTMF events and
    forward digits to Omni; react to Omni events with `uuid_break` (barge-in) and
    `uuid_transfer` (escalate to a human).

    ```js server.js (control plane) theme={null}
    const control = new esl.Connection(FS_HOST, 8021, FS_PASSWORD, () => {
      control.subscribe(["DTMF"]);
    });

    control.on("esl::event::DTMF::*", (e) => {
      const uuid = e.getHeader("Unique-ID");
      const digit = e.getHeader("DTMF-Digit");
      const call = calls.get(uuid);
      if (call) forwardDtmf(call.omni, digit);
    });

    function forwardDtmf(omni, digit) {
      // Localized: exact dtmf frame per the Omni wire protocol reference.
      if (omni.readyState === WebSocket.OPEN) {
        omni.send(JSON.stringify({ type: "dtmf", digit }));
      }
    }

    function handleOmniEvent(evt, uuid) {
      // Omni server -> client frames are keyed on `event` (NOT `type`, your
      // outbound configure/dtmf are `type`-keyed). Switching on `evt.type` is
      // the #1 Omni bug: every case misses and the agent appears mute.
      switch (evt.event) {
        case "hello":
        case "session_started":
        case "configured":
          break;

        case "flush":
        case "barge_in":
          // Barge-in: stop the agent audio currently playing into the channel.
          control.api("uuid_break", uuid);
          break;

        case "transfer_to_human":
          // Hand the live channel off to a human. `destination` comes from the
          // tool's per-agent setting; fall back to your default extension.
          control.api("uuid_transfer", `${uuid} ${evt.destination || process.env.HUMAN_EXTENSION} XML default`);
          break;

        case "send_dtmf":
          control.api("uuid_send_dtmf", `${uuid} ${evt.digits}`);
          break;

        case "play_hold":
          // Play hold audio; it stops when the next agent audio arrives.
          control.api("uuid_broadcast", `${uuid} local_stream://moh aleg`);
          break;

        case "end_call":
          control.api("uuid_kill", uuid);
          break;

        case "collect":
          // Fire-and-forget: the value returns via normal speech (STT) or the
          // DTMF you already forward inbound. Nothing to do for most agents.
          break;

        case "session_end":
          calls.get(uuid)?.fsWS.close();
          break;
      }
    }

    app.listen({ port: 8080, host: "0.0.0.0" });
    ```

    <Info>
      The inbound **event names** (`flush`, `transfer_to_human`, `send_dtmf`,
      `play_hold`, `collect`, `end_call`, `session_end`) are stable (server frames
      are keyed on `event`); your outbound `configure` / `dtmf` frames are keyed on
      `type`. The call-control verbs are emitted by Omni but **executed here**, they
      only happen because this switch handles them. Exact payload fields come from the
      [Omni wire protocol §4.1](/realtime/omni-protocol#4-1-call-control-frames).
      `forwardDtmf` is the only spot that constructs an Omni control frame.
    </Info>
  </Step>
</Steps>

## Run it

```bash theme={null}
npm install
FS_HOST=127.0.0.1 FS_PASSWORD=ClueCon bridge_host=<bridge-host> npm start
```

Reload the dialplan (`fs_cli -x "reloadxml"`), set `bridge_host` to where your
Node process is reachable, then dial extension `9000`. The agent should greet the
caller within a second. Talk over it to confirm `uuid_break` cuts the agent off;
press a DTMF key and watch it reach Omni; trigger a transfer to confirm the
channel reaches the human extension.

## Codec & rate notes

* **No resampling at the fork.** Fork at `16k` and run Omni at `rate=16000`, L16/16 kHz ↔ PCM16/16 kHz is a passthrough. If the caller's leg is 8 kHz,
  FreeSWITCH transcodes it to 16 kHz for the fork (8000 → 16000 is a 2:1
  upsample done inside FreeSWITCH), so Omni always sees a clean 16 kHz stream.
* **Endianness, not rate, is the usual culprit** for distorted audio, see the
  warning above.
* **`park` keeps the channel alive.** Without it the call can tear down before
  the fork is established.

## Troubleshooting

| Symptom                        | Likely cause                          | Fix                                                                               |
| ------------------------------ | ------------------------------------- | --------------------------------------------------------------------------------- |
| Static / white noise both ways | Endian mismatch                       | `buf.swap16()` on the fork audio in both directions                               |
| Agent sounds slow or fast      | Fork rate ≠ Omni rate                 | Fork at `16k` and connect Omni at `rate=16000`                                    |
| Channel hangs up before audio  | Missing `park`                        | Add `<action application="park"/>` after starting the fork                        |
| Agent talks over the caller    | `flush` not wired to `uuid_break`     | Call `uuid_break <uuid>` on every `flush` event                                   |
| DTMF never reaches Omni        | Not subscribed to DTMF, or wrong UUID | `control.subscribe(["DTMF"])`; map by `Unique-ID`                                 |
| Transfer fails                 | Bad extension/context                 | Verify the `uuid_transfer <uuid> <ext> XML <context>` args route in your dialplan |
| Omni WS closes immediately     | Bad key/agent                         | Check the close code in [Errors & limits](/errors-and-limits)                     |
| `module_exists` returns false  | Fork module not loaded                | Load `mod_audio_stream` in `modules.conf.xml` and `reloadxml`                     |

## Next steps

<CardGroup cols={2}>
  <Card title="Omni wire protocol" href="/realtime/omni-protocol">Exact event payloads and close codes.</Card>
  <Card title="Browser voice agent" href="/guides/browser-voice-agent">The same agent in the browser via WebRTC.</Card>
  <Card title="Phone agent with Twilio" href="/guides/twilio-voice-agent">Media Streams bridge at μ-law 8 kHz.</Card>
  <Card title="Errors & limits" href="/errors-and-limits">Rate limits, concurrency, and retries.</Card>
</CardGroup>