Skip to main content
Omni agents can call functions during a live call, look up an order, book an appointment, search your knowledge base, without leaving the voice session. Every tool runs through the same tools[] array and the same soft result contract (a tool failure never breaks the turn). Every tool has an execution mode (returned on GET /v1/tools) that says who runs it: Manage everything from the Tools screen in the console: browse the hosted catalog, register custom tools, see a live call log.

Hosted catalog (zero setup)

Hosted tools need no webhook and no hosting, PyAI runs them for you. Enable one by adding its name to tools[] (or toggling it in Agents → Tools), and the agent’s brain calls it mid-conversation. GET /v1/tools returns the live catalog (each row has "execution": "hosted", an id, a side_effect, and, where relevant, a config_schema).

Catalog at a glance

🔒 = secret (encrypted at rest, returned masked). The nine hosted read tools run today with zero setup. The five engine call-control tools are emitted by the engine today, but a phone-call action only happens if your telephony transport handles the frame, see Engine mode before enabling them. The remaining action tools (send_sms, send_email, calendar, payment) are reserved catalog entries, the name, execution, side_effect, input_schema, and config_schema are stable so you can build against them now, but calling one returns a soft {"error":"hosted_tool_unavailable"} until it ships, and (where it needs settings) until those settings are saved on the agent (see Tool settings).

Live hosted tools, arguments & results

Arguments are what the brain fills in per call; results are returned to the brain (and never break the turn, a bad argument comes back as a soft error).
Searches the knowledge bases bound to the agent (falls back to the org’s default KBs for zero-state sessions). Knowledge stays customer-hosted.
  • query (string, required), what to look up.
  • top_k (integer, optional, default 5, 1-20), how many passages.
Safe arithmetic only: + - * / % ^, parentheses, unary ±, decimals and exponent notation. No identifiers or function calls (nothing to inject).
  • expression (string, required, ≤200 chars), e.g. "2 + 3 * 4".
  • timezone (string, optional, default UTC), an IANA name like America/New_York.
Converts within a dimension: length (mm cm m km in ft yd mi), mass (mg g kg oz lb), or temperature (c f k).
  • value (number, required), from (string, required), to (string, required).
Live mid-market rates.
  • from (string, required), to (string, required), 3-letter ISO codes.
  • amount (number, optional, default 1).
Geocodes the place name, then returns current conditions and a 3-day forecast.
  • location (string, required), city/place name.
  • units (string, optional, metric | imperial, default metric).
  • address (string, required), place/address to resolve.
  • limit (integer, optional, default 5, 1-10).
Backed by Wikipedia today (no key, no setup).
  • query (string, required).
  • limit (integer, optional, default 5, 1-10).
  • topic (string, required).
  • limit (integer, optional, default 5, 1-10).

Tool settings (per-agent config)

Some tools need a little setup before they can run, an action tool like send_sms needs a from-number and a provider key; transfer_to_human needs a destination. A tool declares what it needs in its config_schema (returned by GET /v1/tools), and you supply the values per agent on the tool binding’s config. This keeps the same tool reusable across agents with different settings.
Render those fields as a form (the console Agents → Tools tab does this for you), then save the answers on the binding:
Fields marked "secret": true are encrypted at rest and returned masked ("********") on every read, re-send the mask (or leave the field blank) to keep the stored value, or send a new value to rotate it. PyAI decrypts a secret only at execution time. Custom tools can declare their own config_schema too, or omit it and let your webhook manage its own configuration.

Server mode (PyAI calls your webhook)

Register a tool with a webhook_url. When the agent calls it, PyAI validates the model’s arguments, then calls your endpoint from a dedicated, egress-isolated service (never from the model) under a hard timeout, and feeds the result back to the agent.
Save the returned hmac_secret (whsec_…), it is shown once and is how you verify calls are genuinely from PyAI. Optionally set auth_header + auth_secret (stored encrypted, injected only into the outbound call) if your webhook needs its own bearer token.
Re-syncing is idempotent. POST /v1/tools upserts on (org, name): posting a tool whose name your org already has updates it in place (HTTP 200, no duplicate, hmac_secret preserved) instead of creating a second copy, so a multi-tenant deploy can re-push its tool catalog safely. A brand-new name creates the tool (HTTP 201, hmac_secret returned once). You can also update explicitly by id with POST /v1/tools/{id}.Rotating the signing secret with no dropped calls: call POST /v1/tools/{id} with { "rotate_secret": true } to mint a new hmac_secret (returned once). For a zero-drop rotation, deploy verification that accepts both the old and new secret, rotate, confirm traffic verifies against the new secret, then drop the old one.
Bind tools to an agent profile (or just list them in the configure frame):

What PyAI sends your webhook

A POST with a JSON body:
and these headers: The signature is HMAC-SHA256(secret, "<t>.<raw_body>") in hex. Verify it on every call and reject anything where the timestamp is stale (e.g. > 5 min) to defeat replays.

What your webhook should return

Return JSON, any shape the agent can use. It is handed back to the model as the tool result:

Reliability you get for free

  • Soft-fail, a timeout, 5xx, or unreachable webhook becomes a structured error the agent can apologize for; it never crashes the call.
  • Timeouts & size cap, the effective budget is min(timeout_ms, platform ceiling); the ceiling is 5 s by default, so registering timeout_ms above it has no effect unless PyAI raises the ceiling for your org. Results larger than ~6 KB are truncated.
  • Idempotency, a retried voice turn reuses the same call_id; PyAI replays the prior successful result instead of calling your webhook again, so an action tool (charge a card, send a text) fires once. Still, treat call_id as an idempotency key on your side.
  • Circuit breaker, if your webhook fails repeatedly, PyAI briefly stops calling it (and tells the agent the tool is unavailable) instead of hammering a broken endpoint and slowing every turn.
  • SSRF-safe, only the exact webhook_url you registered is ever called, and only over public HTTPS (private/loopback/metadata addresses are rejected).

Engine mode (call control)

Call-control tools (transfer_to_human, send_dtmf, play_hold, collect, end_call) are media/SIP actions on a phone call. The Omni engine decides when to fire one and emits a control frame; the actual telephony action runs in the transport that bridges Omni to the carrier, Twilio Media Streams, FreeSWITCH, your SIP stack. There are two ways to run that transport:
  • PyAI managed Telephony (beta), buy a number, bind it to an agent, and PyAI runs the bridge. Managed call control is coming soon: PyAI’s bridge will perform these verbs for you (and fold the agent’s destination setting into transfer_to_human), so they work end-to-end with no transport code.
  • Your own transport, you connect the WebSocket and translate each frame into a carrier operation (the Twilio and FreeSWITCH guides show working handlers).
On a self-hosted transport, enable a call-control tool only once your transport handles its frame. If the agent calls transfer_to_human but your bridge ignores the frame, the agent will say it’s transferring while the call stays put. (On managed Telephony with managed call control, this is handled for you.)
What you implement: on each 0x03 control frame whose event is the tool name, perform the carrier action with the arguments spread in the frame. The exact shapes are in the wire protocol §4.1, and the FreeSWITCH and Twilio guides show working handlers for all five verbs. Quick map: Not running telephony (a browser or in-app agent)? These verbs are phone concepts and generally don’t apply, transfer_to_human and friends assume a call leg. Use server tools for app actions instead.

Client mode (on the WebSocket)

For browser apps already running your code on the socket, omit webhook_url (or set execution: "client"): the engine emits a tool_call frame and you reply with a tool_result on the same socket. See the wire protocol.

Side effects & confirmation

Mark tools that change state with "side_effect": "action" (vs. "read"). The agent is prompted to confirm an action tool with the caller before firing it.

See also

Omni wire protocol

Full frame reference including tool_call / tool_result.

Browser voice agent

End-to-end website agent with grounding.

API reference

/v1/tools and /v1/agents/{id}/tools.

Tools in the console

Catalog, custom-tool builder, and call log.