> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pyai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Live Translation with Twilio

> Connect two human Twilio calls to one PyAI interpretation session using bidirectional Media Streams.

Build a media bridge with one bidirectional Twilio stream per person and one
PyAI interpretation WebSocket. The bridge decodes each microphone stream,
routes it to the correct participant and plays translated speech to the
opposite participant.

> [!NOTE]
>
> The SDK provides the two-party translation room and Twilio media transport.
> Your server remains responsible for owning both calls, validating Twilio's
> signed WebSocket upgrade, storing transcripts and call events, and allowlisting
> internal contacts. Bounded synthetic browser-call tests passed for Hindi ↔ English
> and Punjabi ↔ English, including both directions, echo blocking, saved transcripts
> and replacement-agent transfers. Validate your own call setup and human speech
> quality before production.

Read the [shared architecture and PyAI protocol](/guides/live-translation-telephony.md)
first. Keep your Twilio credentials and live `interpret:session` key on the
bridge server.

The repository includes a bounded
`twilio-live-translation` test host
for dedicated customer, agent and replacement-agent browser calls. Build the
current recipe and SDK from the same repository revision. The host validates
webhook signatures, issues single-use participant links, persists transcripts
to private storage and applies carrier time limits. Tests used real Twilio calls
and the deployed PyAI API; they did not certify PSTN routing, physical speaker
audibility or JustCall integration.

## Recordings and transcripts in Console

Open [Live Translation in Console](https://console.pyai.com/live-translation),
select your team's live project, and start a new conversation or call. Confirm
that everyone agrees to recording before starting. For a carrier call, the
Console explicitly shows whether the connected host supports recording.

Choose **End conversation** or **End call** and wait for saving to finish. In
**Call history**, select the call to play its recording, download the audio, or
read and download the original and translated transcripts. Other signed-in
members of the same project can open the same history. Invitation links and
upload tokens do not grant access to stored recordings.

Original audio is on the left channel and translated audio is on the right.
Browser recordings contain the microphone and the translation rendered by the
browser. Carrier recordings contain source audio accepted by the bridge and
translated audio sent to the call; a cancelled or interrupted translation can
sound different at the listener. A recording is not proof that someone heard
all sent audio. Agent replacement retains the same call record and transcript.

Earlier tests have no recoverable recording. An incomplete upload is labelled
**Incomplete**, and any successfully saved transcript remains available.
Recordings stay private; playback and download require current project access.

## Try two-person calling in Console

The [internal Twilio rehearsal](https://console.pyai.com/live-translation#twilio-preview)
requires a verified saaslabs.co account and an active live project. Choose
Hindi ↔ English or Punjabi ↔ English, create a room, and give the customer and
agent their separate invitations. Each person should use a separate device and
headphones. Original speech and translated captions appear in the operator view;
**Invite replacement agent** tests an internal transfer without replacing the
customer's call. Finish with **End call** and wait for cleanup confirmation.

Console displays the current preview's expiry and allowance. This bounded
internal rehearsal does not establish ten-call capacity or customer billing.
Keep invitations private; they are short-lived and single-use.

## Connect the customer call

Return this TwiML from your authenticated answer webhook. The WebSocket URL
belongs to **your bridge**, not the PyAI API:

```xml theme={null}
<Response>
  <Connect>
    <Stream url="wss://bridge.example.com/twilio/media/one-time-server-token" />
  </Connect>
</Response>
```

Create a separate outbound call to the human agent, with a server-controlled
answer webhook returning the same structure with `leg=agent`. Pair both calls
in trusted server state. The SDK's `TwilioStreamAdmission` issues a short-lived,
single-use path and validates `X-Twilio-Signature`; use that binding together
with `start.accountSid`, `start.callSid` and `start.streamSid` before creating
the media transport.

Twilio supports one bidirectional stream per call and delivers its inbound
track. `<Connect><Stream>` blocks following TwiML while connected. A
`<Start><Stream>` or ordinary conference audio tap cannot supply the required
listener-specific bidirectional output. See
[Media Streams](https://www.twilio.com/docs/voice/media-streams) and
[Stream parameters](https://www.twilio.com/docs/voice/twiml/stream).

## Authenticate and convert the audio

Validate `X-Twilio-Signature` on webhooks and the WebSocket upgrade using
Twilio's supported validation library and the correct public request URL behind
your proxy. Bind the expected account, `start.callSid`, `start.streamSid` and
room/leg to the authenticated call record. Stream URLs do not support query
parameters; use nested `Parameter` elements for pairing metadata.

Twilio media uses base64 headerless μ-law at 8 kHz. Decode and statefully
resample to PyAI's 16 kHz PCM16 input. Convert output back to μ-law at 8 kHz,
with bounded, paced queues. The existing `@pyai/twilio` codec and streaming
resampler helpers are used by `TwilioInterpretationTransport`; pair it with
`InterpretationRoom` and a durable `record` callback.

On the intended listener's stream, send:

```json theme={null}
{"event":"media","streamSid":"target-stream-sid","media":{"payload":"<base64 raw mulaw8k>"}}
```

## Map marks without inventing playback

After the final media for an utterance, send an identity-bound mark:

```json theme={null}
{"event":"mark","streamSid":"target-stream-sid","mark":{"name":"unique-playback-boundary"}}
```

Track its stream, participant epoch, output generation, utterance and original
PyAI sample count. A returned current mark may confirm complete playback only
if that boundary was not invalidated by a clear.

For cancellation, invalidate pending marks **before** stopping local output and
sending:

```json theme={null}
{"event":"clear","streamSid":"target-stream-sid"}
```

Twilio also returns pending marks for audio discarded by `clear`. Those marks
must never become `status: "played"`. See
[Twilio's mark and clear semantics](https://www.twilio.com/docs/voice/media-streams/websocket-messages).

Twilio has no distinct documented clear acknowledgement. The SDK sends a fresh
sentinel mark after clear on the same ordered socket and waits for that barrier
before reporting `playback.cleared`. Carrier transfer tests returned the barrier
and resumed the replacement participant. Unit tests separately reject invalidated
old marks; the carrier tests did not naturally produce a late stale mark.

## Handle transfers and hangup

Perform transfers with Twilio call control and the replacement sequence in the
[shared guide](/guides/live-translation-telephony.md). Clear both playback queues
and fence the previous agent's call/stream before admitting the new agent.
DTMF on bidirectional Media Streams is inbound-only; keep outbound DTMF in your
call-control design.

When either call ends, apply your room's waiting/hangup policy, send
`session.end` and close the PyAI socket promptly. Do not replay uncertain
speech after reconnect. Validate both translation directions, late marks after
clear, echo blocking and a real transfer before enabling customer traffic.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.