Skip to main content
​
Hear streaming: session start no longer depends on the billing plane; explicit capacity signalling
  • Streams start even when our billing plane is unavailable. Opening a Hear stream used to wait on an internal usage reservation before any transcription began, and a hiccup in that service refused the stream. The reservation is now recorded locally on the streaming node and reconciled in the background, so a billing-plane blip no longer affects session start or end. Metering is unchanged.
  • Capacity is signalled up front. The streaming edge now reads the speech engine’s live stream count and, when the engine is at capacity, refuses a new stream immediately with error code capacity and close code 1013 (retry) rather than accepting it into a degraded state. This is the same retryable signal introduced on 2026-09-03 for mid-call shedding.
​
Docs: first-call loops, SDKs page, expected responses
  • Quickstart now closes every first call. GET /v1/me shows a sandbox identity (including scopes the mint does not grant). Hear has curl / Python / Node tabs and a sample { "text": "..." }. Omni documents the configured ack. Cast preview is runnable with stock_dorit_en_us. Recap has a submit-and-read path with a complete recap.record.v1 sample.
  • Clone examples match the shipped SDKs. Use pyai.clones, not pyai.voice.clones. Poll GET /v1/voice/clones (there is no GET-by-id).
  • New SDKs page in Get started. Auth and Errors now show the first-call 401 invalid_api_key / 403 insufficient_scope / 402 credit_exhausted envelopes.
​
Hear streaming: finals on long turns, and no more mid-call drops under load
  • A final now arrives at least every 15 seconds of audio. Two-party call audio that never pauses for endpointing_ms used to run one open utterance for minutes and, on a dropped socket, never produce a final at all. The engine now closes such an utterance at 15 s with endpoint_reason: "max_utterance" and starts a fresh utterance_id; pipelines that store only finals get every word. Silence endpoints and commit behave exactly as before.
  • Mid-call engine drops no longer end your session. If the speech engine behind the stream closes or stalls, the stream re-establishes its engine leg and resumes; you may see a gap of up to one chunk, not a closed socket. When the platform is genuinely out of capacity you get error code capacity and close code 1013 (retry), instead of internal: stream terminated unexpectedly.
  • Concurrency: streams are now decoded together on the GPU rather than one at a time, and each engine sheds load with a retryable 1013 instead of degrading every active stream. Measured limits are published with the rollout record.
​
Hindi word-level timestamps
  • Hindi joins word-level timestamps on async transcription. words[] now carries per-word times for hi alongside en, es, fr, de, it, and pt — native Devanagari, no romanization. Hindi timing precision is a step below the Latin-script languages on heavily code-switched telephony (roughly four in five word starts within 120 ms); segments remain robust either way. nl keeps turn-level segments with an empty words array.
​
Word-level timestamps on async transcription
  • words[] is back on async transcription jobs — now word-granular. A forced-alignment pass restores per-word start/end times for en, es, fr, de, it, and pt (median word-start accuracy measured at 20-50 ms on real telephony). segments are now built from word timings on these languages; hi and nl keep turn-level segments with an empty words array for now.
  • Word timings survive number formatting: tokens rewritten by digit formatting carry their span’s true start/end rather than a guess.
  • No request change needed — jobs that always requested timestamps simply start receiving them.
​
Async Trace live: deterministic PII scan on transcription jobs
  • trace: true on async transcription jobs now runs a real scan. The deterministic detector set from the Trace inline gate — SSN, card numbers (grouped and contiguous), CVV in context, email, US phone — runs over the final transcript; findings are redacted as [REDACTED_<TYPE>] and the completed job carries the redacted transcript plus a trace summary (verdict, PII count). Requires the Trace entitlement; bills one Trace call per job. rule_pack.entities narrows the scan to a subset.
  • Patterns run on the formatted transcript: keep the default numerals (digits) for full effect — verbatim transcripts can leave spoken-out numbers unredacted. Not yet combined with diarize/channel.
  • This replaces yesterday’s temporary 400 on async trace.
​
API reference: Clone, Hear, and Speak
  • The generated API reference matches the docs sidebar. OpenAPI tags are now Speech To Text (Hear), Text To Speech (Speak), and Clone. Clone list/create/delete live under Clone, not Speak. Voice design stays on Speak.
​
Docs nav: Clone, Hear, and Speak
  • Clone is its own docs category. Voice enrollment no longer lives under Speak. Sidebar groups are now Speech To Text (Hear), Text To Speech (Speak), Clone, and Cast.
  • Voice design is documented as live. The Clone guide no longer calls /v1/voice/design a rolling-out surface.
​
Async diarization restored; async Trace paused; Cast voice roster
  • Channel and diarize segments restored on async jobs. Between 2026-08-26 and 2026-09-02, channel jobs returned one block per speaker instead of conversational turns, and mono diarize jobs fell back to a flat transcript. Turn segmentation is now derived from the audio itself and no longer depends on engine word timings; segments carry per-turn speakers and timing again. The words array remains empty on async jobs until word-level alignment returns to the batch engine.
  • Async trace is temporarily rejected. The current batch engine cannot run the Trace scan, and a job that requests PII scanning must never complete without one — trace: true on async submission now returns 400 instead of silently skipping the scan. The synchronous Trace surface is unaffected.
  • Cast voices are the capabilities roster. GET /v1/cast/capabilities is the authoritative voice list, as documented; legacy undocumented names from before the 2026-08-31 engine migration (e.g. sarah) now return 422 unsupported_voice.
​
New Hear recognition engine
  • Hear is served by a new speech-recognition engine across streaming and async batch. In a blind side-by-side judged on real calls, the new engine’s transcripts were preferred in 87.3% of disagreements with the prior engine.
  • Async batch jobs now transcribe all eight Hear languages, with the spoken language auto-detected per call. The detected language code is not yet returned in the result.
​
Vocabulary biasing across Hear and Omni
  • Per-job vocabulary: async transcription jobs accept an optional vocabulary list of up to five distinctive terms, sanitized and applied to that job only.
  • Organization vocabulary: stored Hear vocabulary settings with explicit batch and streaming activation; streaming biasing uses the same bounded sanitizer.
  • Managed Agent vocabulary: Omni Agent profiles accept an opt-in vocabulary list, frozen at session start and never inherited from organization settings.
  • Vocabulary biases recognition toward your terms; it does not choose the transcription language or steer the output script.
​
200 ms first partial on streaming
  • Hear streaming publishes ~200 ms to first partial measured on real speech in-region. Partials remain revisable; finals are the stable text to commit.
​
Eight-language sync and streaming STT
  • Hear sync and streaming serve eight languages: en, es, fr, de, hi, it, pt, and nl via the explicit language parameter. Omission means English; codes outside the set are rejected with 400 unsupported_language, never silently mis-transcribed.
​
Certified browser widget v11
  • Widget v11 is the certified browser widget for opening playback and Omni protocol compatibility. New embeds should load https://cdn.pyai.com/widget/v11/pyai-widget.js.
  • Versioned widget URLs are immutable. Sites pinned to v1-v10 should update their script URL to v11.
​
Website knowledge crawl
  • Hosted knowledge bases can crawl a public website. POST /v1/knowledgebases/{id}/crawls discovers same-origin pages, ranks them, and registers each selected URL as a normal document. Poll until indexed before you treat the site as ready. The agent wizard crawls the first website you add.
​
Docs paths and website widget
  • Choose your path is a real docs page. The navbar button no longer jumps straight to console signup.
  • Create agents via API collects profile CRUD, hosted KB bind, tools, and session_label in one guide.
  • Frameworks (LiveKit, Pipecat, Cursor) sit in their own nav group, not under CRM integrations. Omni v2 migration moved to the bottom of Omni.
  • Hosted knowledge bases are the default grounding path. kb_endpoint remains for teams that already run retrieval.
  • Website widget docs match the console tab name: Website & phone.
​
Omni v2 chat URL discontinued
  • Hindi is live on Omni with four Standard-tier voices. Choose hi1 through hi4; omit voice_id for the default. See the pricing page for current tier treatment.
  • Short multilingual voice aliases: es1–es3, fr1, de1, and hi1–hi4. Aliases are permanent convenience inputs; the configured.voice_id ack reports the canonical stock id actually served.
  • The old Omni v2 chat URL has been discontinued. It now returns 404. Connect to wss://api.pyai.com/v1/omni?format=pcm16&rate=24000.
  • Query migration: move the value of agent or agent_id to session_label. The label is optional. Remove model. Replace an access_token query parameter with the pyai-key.<key> WebSocket subprotocol or api_key=<key>.
  • Auth: browser clients should send Sec-WebSocket-Protocol: pyai-key.pyai_live_.... Server clients can use Authorization: Bearer <key>, x-api-key: <key>, or the api_key query fallback. The key needs omni:session.
  • Configure: after the upgrade, send 0x03 || {"type":"configure",...} as a binary frame. Client control JSON uses type; server JSON uses event.
  • Audio and control framing: prefix caller and agent PCM16 with 0x01, transcripts with 0x02, and control frames with 0x03. Bare PCM and text control frames are not accepted.
  • Behavioral differences: the discontinued URL returns HTTP 404. Canonical query and auth failures can reject the HTTP upgrade with 400/401/403; post-upgrade closes include 4401, 4403, 4429, and 1011.
  • Follow the complete migration guide for before-and-after URLs, code, close handling, and troubleshooting.
​
AMD, answering machine detection
  • PyAI AMD is live, answering-machine detection that tells your dialer who or what answered a call (human, voicemail, IVR, live receptionist, iPhone/Google screening, dead number, fax) in a fraction of Twilio’s dead-air dwell, with the word-level reason it decided.
  • One-line-TwiML Twilio drop-in: point a <Start><Stream> at wss://api.pyai.com/v1/amd/stream and drop machineDetection. <Start> forks the audio, so your call flow continues untouched. The answered_by_twilio field maps to Twilio’s exact AnsweredBy enum, so your routing logic doesn’t change.
  • One aggressiveness dial (0-1, per account via POST /v1/amd/config or per call via a TwiML <Parameter>): human-safe ↔ machine-aggressive.
  • SDKs: pyai.amd.config / pyai.amd.calls (+ amd.stream() in TS); MCP tools configure_amd, list_amd_calls, get_amd_call. New guide: Answering machine detection.
  • Metering: AMD records answered-call usage. See the pricing page for current terms.
​
Post-call data capture + Omni tools GA
  • Post-call data capture, declare an extraction_schema + extraction_webhook_url on an agent and PyAI extracts structured JSON from each call’s transcript and delivers it signed (omni.call.extracted). No client or engine code. New guide: Post-call data capture; configure it in the console Agents → Data capture tab.
  • npm create pyai-app@latest scaffolds any runnable example in one command, no clone, no setup. Browse them at the examples repo.
  • New examples, each linked from its guide and exercised in nightly CI: OpenAI drop-in, voice cloning, call analytics, and offline omni-mock.
  • Machine-readable Omni contract: AsyncAPI 3.0 at contracts/omni-asyncapi.yaml, see Omni wire protocol.
  • Omni function calling (tools) is GA, client-loop tool_call / tool_result on the WebSocket. New guide: Omni tools. REST: /v1/tools, /v1/agents/{id}/tools.
  • tools[] moved from roadmap → live in the protocol reference (~5 s default timeout).
​
One docs home + drift guards
  • docs.pyai.com is now the single home for everything: quickstart, guides, and the generated API reference. api.pyai.com/docs redirects here; api.pyai.com/openapi.json remains the machine-readable contract.
  • Quickstart verifies your key first. A new GET /v1/me step gives you a self-diagnosing first 200 in seconds before any audio call.
  • Docs can’t drift to dead ids. CI now fails if any sample uses a voice or model id that isn’t in the live catalog, every snippet stays copy-paste-runnable.
  • Example voice standardized to stock_dorit_en_us across the quickstart, guides, and SDK READMEs.
​
Docs accuracy & unified API reference
  • REST reference unified in-site. The full OpenAPI reference now lives inside these docs under the API reference tab (generated from https://api.pyai.com/openapi.json), no more bouncing to a separate page.
  • Hear streaming contract: pyai-hear-v1 emits the canonical bare frames (partial, partial_stable, speech_final, final, usage, error) and force-finalizes with the JSON frame {"type":"commit"}. Cue is unavailable on the serving route. See Stream speech-to-text for current behavior.
  • Telephony output on Speak: request g711_ulaw / g711_alaw for native 8 kHz μ-law/A-law, no client-side resampling. New reference: Telephony audio with the exact integer resample ratios for the cases where you still convert yourself.
​
Developer experience
  • New guides: browser voice agent, Twilio phone agent, and FreeSWITCH.
  • Official SDKs: @pyai/sdk (TypeScript/JavaScript) and pyai (Python), plus the pyai CLI smoke tester.
  • Idempotency-Key support and cursor pagination on transcription jobs.
  • Machine-readable error codes documented end to end (see Errors & limits).
  • Sandbox tier: pyai_test_ keys work instantly with no billing gate.