Use PyAI with LiveKit Agents

Use LiveKit Agents when you want to own the realtime worker, media rooms, LLM, tools, and turn policy. Add PyAI Hear and PyAI Speak as the speech components. If you would rather run the complete voice agent through one managed connection, use Omni.

The PyAI plugin is maintained by PyAI and currently supports LiveKit’s Python Agents SDK. LiveKit also supports Node.js, but the package in this guide does not.

What stays under your control

LiveKit keeps the room, media transport, worker lifecycle, VAD, LLM, tools, and application logic. PyAI handles streaming transcription and speech synthesis. Your existing LiveKit deployment model does not change.

Prerequisites

Python 3.10 or newer
A LiveKit Agents worker
A PyAI key with hear:stream, hear:transcribe, and speak:synthesize scopes
A voice id from GET https://api.pyai.com/v1/voices

A pyai_test_ sandbox key includes the required Hear and Speak scopes, starts working immediately, and does not require billing. Keep all API keys on the server-side worker.

Install

Install the verified public release from PyPI:

pip install livekit-plugins-pyai==0.1.0

Set the key once:

export PYAI_API_KEY=pyai_test_...

Replace the speech components

Keep the LLM and tools your worker already uses. Change only the stt and tts values in AgentSession:

from livekit.agents import AgentSession
from livekit.plugins import pyai

# `llm` and `vad` are the objects your existing worker already creates.
session = AgentSession(
    stt=pyai.STT(language="en"),
    llm=llm,
    tts=pyai.TTS(voice="stock_emma_en_gb"),
    vad=vad,
)

That is the integration. LiveKit pushes caller audio into pyai.STT, receives interim and final transcripts, runs your LLM and tools, then streams the generated text through pyai.TTS back into the room.

Worker function

Drop this function into the worker that already creates your LLM and VAD:

from livekit.agents import Agent, AgentSession, JobContext
from livekit.plugins import pyai


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="You are a concise phone support agent."
        )


async def run_agent(ctx: JobContext, llm, vad) -> None:
    await ctx.connect()

    session = AgentSession(
        stt=pyai.STT(language="en"),
        llm=llm,
        tts=pyai.TTS(voice="stock_emma_en_gb"),
        vad=vad,
    )
    await session.start(agent=Assistant(), room=ctx.room)

Call run_agent(ctx, llm, vad) from your existing entrypoint. Keep the same WorkerOptions and process command you already use.

Configuration

Hear speech-to-text

pyai.STT(
    api_key=None,                       # defaults to PYAI_API_KEY
    base_url="https://api.pyai.com",
    model="pyai-hear",
    language="en",
    sample_rate=16000,
)

Hear streaming is English-only. The plugin resamples incoming LiveKit frames to 16 kHz and maps PyAI partials and finals to LiveKit transcript events. Its batch recognize() fallback uses the synchronous transcription endpoint.

Speak text-to-speech

pyai.TTS(
    voice="stock_emma_en_gb",
    api_key=None,                       # defaults to PYAI_API_KEY
    base_url="https://api.pyai.com",
    model="pyai-speak",
    sample_rate=24000,
)

Speak returns raw mono PCM at the session rate. Use a streaming-capable stock, cloned, or designed voice id available to the same PyAI organization.

Run and verify

Start the worker

Run the same LiveKit worker command you use today.

Join a room

Connect a browser, phone, or synthetic participant and speak one short sentence.

Check both speech directions

Confirm a final caller transcript reaches the worker and agent audio reaches the room. Then send a second turn on the same session to verify that finalization and interruption handling keep working.

Measure your complete path

Record transport region, worker region, LLM, VAD settings, voice, and PyAI region with any latency result. The framework path is the sum of every component.

Production notes

Keep the LiveKit worker and PyAI speech endpoints in nearby regions.
A live Hear socket counts against the PyAI key’s realtime concurrency.
Hear and Speak meter against your PyAI account. LiveKit Cloud, telephony, your LLM, and worker hosting remain separate.
Do not expose a long-lived PyAI key to a browser or room participant.
Use Omni instead when you want PyAI to own turn-taking, reasoning, tools, caller continuity when an Agent profile has continuity enabled and a caller key is available, and optional managed telephony as one system.

Troubleshooting

Symptom	Check
`ValueError: A PyAI API key is required`	Set `PYAI_API_KEY` on the worker or pass `api_key=`.
`403 forbidden`	Add `hear:stream`, `hear:transcribe`, or `speak:synthesize` to the key.
Caller audio arrives but no final transcript appears	Confirm the worker is current, sends VAD flushes, and keeps the session open for the returned final.
Speech sounds too fast or too slow	Keep the LiveKit output rate and `pyai.TTS(sample_rate=...)` aligned.
New sessions receive `429`	Close idle Hear streams or wait for realtime concurrency to free up.

Which path should I choose?

Choose LiveKit + PyAI for WebRTC rooms, video or multi-participant media, component swapping, and worker-level control. Choose Omni for a phone-first agent where one managed realtime contract is more valuable than owning each stage.

LiveKit Agents documentation

Review LiveKit’s worker, room, telephony, and deployment model.

PyAI Omni overview

Compare the managed speech-to-speech path.

Get started

Omni

Guides

Frameworks

Integrations

Use cases

Reference

Use PyAI with LiveKit Agents

What stays under your control

Prerequisites

Install

Replace the speech components

Worker function

Configuration

Hear speech-to-text

Speak text-to-speech

Run and verify

Production notes

Troubleshooting

Which path should I choose?

LiveKit Agents documentation

PyAI Omni overview

​What stays under your control

​Prerequisites

​Install

​Replace the speech components

​Worker function

​Configuration

​Hear speech-to-text

​Speak text-to-speech

​Run and verify

​Production notes

​Troubleshooting

​Which path should I choose?

LiveKit Agents documentation

PyAI Omni overview

What stays under your control

Prerequisites

Install

Replace the speech components

Worker function

Configuration

Hear speech-to-text

Speak text-to-speech

Run and verify

Production notes

Troubleshooting

Which path should I choose?