> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Use PyAI with LiveKit Agents

> Keep LiveKit rooms and orchestration while using PyAI Hear for streaming speech-to-text and PyAI Speak for agent audio through a Python plugin.

Use **LiveKit Agents** when you want to own the realtime worker, media rooms,
LLM, tools, and turn policy. Add **PyAI Hear** and **PyAI Speak** as the speech
components. If you would rather run the complete voice agent through one
managed connection, use [Omni](/guides/omni-overview).

<Note>
  The PyAI plugin is maintained by PyAI and currently supports LiveKit's Python
  Agents SDK. LiveKit also supports Node.js, but the package in this guide does
  not.
</Note>

## What stays under your control

```mermaid theme={null}
flowchart LR
  caller[Caller] --> room[LiveKit room]
  room --> hear[PyAI Hear STT]
  hear --> worker[Your LiveKit worker]
  worker --> brain[Your LLM and tools]
  brain --> speak[PyAI Speak TTS]
  speak --> room
```

LiveKit keeps the room, media transport, worker lifecycle, VAD, LLM, tools, and
application logic. PyAI handles streaming transcription and speech synthesis.
Your existing LiveKit deployment model does not change.

## Prerequisites

* Python 3.10 or newer
* A LiveKit Agents worker
* A PyAI key with `hear:stream`, `hear:transcribe`, and `speak:synthesize`
  scopes
* A voice id from `GET https://api.pyai.com/v1/voices`

<Tip>
  A `pyai_test_` sandbox key includes the required Hear and Speak scopes,
  starts working immediately, and does not require billing. Keep all API keys
  on the server-side worker.
</Tip>

## Install

Install the verified public release from PyPI:

```bash theme={null}
pip install livekit-plugins-pyai==0.1.0
```

Set the key once:

```bash theme={null}
export PYAI_API_KEY=pyai_test_...
```

## Replace the speech components

Keep the LLM and tools your worker already uses. Change only the `stt` and
`tts` values in `AgentSession`:

```python theme={null}
from livekit.agents import AgentSession
from livekit.plugins import pyai

# `llm` and `vad` are the objects your existing worker already creates.
session = AgentSession(
    stt=pyai.STT(language="en"),
    llm=llm,
    tts=pyai.TTS(voice="stock_emma_en_gb"),
    vad=vad,
)
```

That is the integration. LiveKit pushes caller audio into `pyai.STT`, receives
interim and final transcripts, runs your LLM and tools, then streams the
generated text through `pyai.TTS` back into the room.

## Worker function

Drop this function into the worker that already creates your LLM and VAD:

```python theme={null}
from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import pyai


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a concise phone support agent."
        )


async def run_agent(ctx: JobContext, llm, vad) -> None:
    await ctx.connect()

    session = AgentSession(
        stt=pyai.STT(language="en"),
        llm=llm,
        tts=pyai.TTS(voice="stock_emma_en_gb"),
        vad=vad,
    )
    await session.start(agent=Assistant(), room=ctx.room)
```

Call `run_agent(ctx, llm, vad)` from your existing entrypoint. Keep the same
`WorkerOptions` and process command you already use.

## Configuration

### Hear speech-to-text

```python theme={null}
pyai.STT(
    api_key=None,                       # defaults to PYAI_API_KEY
    base_url="https://api.pyai.com",
    model="pyai-hear",
    language="en",
    sample_rate=16000,
)
```

Hear streaming is English-only. The plugin resamples incoming LiveKit frames to
16 kHz and maps PyAI partials and finals to LiveKit transcript events. Its batch
`recognize()` fallback uses the synchronous transcription endpoint.

### Speak text-to-speech

```python theme={null}
pyai.TTS(
    voice="stock_emma_en_gb",
    api_key=None,                       # defaults to PYAI_API_KEY
    base_url="https://api.pyai.com",
    model="pyai-speak",
    sample_rate=24000,
)
```

Speak returns raw mono PCM at the session rate. Use a streaming-capable stock,
cloned, or designed voice id available to the same PyAI organization.

## Run and verify

<Steps>
  <Step title="Start the worker">
    Run the same LiveKit worker command you use today.
  </Step>

  <Step title="Join a room">
    Connect a browser, phone, or synthetic participant and speak one short
    sentence.
  </Step>

  <Step title="Check both speech directions">
    Confirm a final caller transcript reaches the worker and agent audio reaches
    the room. Then send a second turn on the same session to verify that
    finalization and interruption handling keep working.
  </Step>

  <Step title="Measure your complete path">
    Record transport region, worker region, LLM, VAD settings, voice, and PyAI
    region with any latency result. The framework path is the sum of every
    component.
  </Step>
</Steps>

## Production notes

* Keep the LiveKit worker and PyAI speech endpoints in nearby regions.
* A live Hear socket counts against the PyAI key's realtime concurrency.
* Hear and Speak meter against your PyAI account. LiveKit Cloud, telephony, your
  LLM, and worker hosting remain separate.
* Do not expose a long-lived PyAI key to a browser or room participant.
* Use Omni instead when you want PyAI to own turn-taking, reasoning, tools,
  caller continuity when an Agent profile has continuity enabled and a caller
  key is available, and optional managed telephony as one system.

## Troubleshooting

| Symptom                                              | Check                                                                                                |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `ValueError: A PyAI API key is required`             | Set `PYAI_API_KEY` on the worker or pass `api_key=`.                                                 |
| `403 forbidden`                                      | Add `hear:stream`, `hear:transcribe`, or `speak:synthesize` to the key.                              |
| Caller audio arrives but no final transcript appears | Confirm the worker is current, sends VAD flushes, and keeps the session open for the returned final. |
| Speech sounds too fast or too slow                   | Keep the LiveKit output rate and `pyai.TTS(sample_rate=...)` aligned.                                |
| New sessions receive `429`                           | Close idle Hear streams or wait for realtime concurrency to free up.                                 |

## Which path should I choose?

Choose **LiveKit + PyAI** for WebRTC rooms, video or multi-participant media,
component swapping, and worker-level control. Choose **Omni** for a phone-first
agent where one managed realtime contract is more valuable than owning each
stage.

<CardGroup cols={2}>
  <Card title="LiveKit Agents documentation" href="https://docs.livekit.io/agents/">Review LiveKit's worker, room, telephony, and deployment model.</Card>
  <Card title="PyAI Omni overview" href="/guides/omni-overview">Compare the managed speech-to-speech path.</Card>
</CardGroup>