Crawl a public website into a knowledge base
Discover same-origin pages from a public seed URL (robots.txt / sitemap, then a shallow link walk), rank them, and register each selected page as a normal URL document. Private, loopback, and metadata addresses are rejected. Defaults to 25 pages, hard-capped at 40. Already-ingested URLs in this knowledge base are reused. Poll document status until indexed before treating the site as ready. Requires the kb:manage scope.
Authorizations
Use Authorization: Bearer pyai_live_... (or pyai_test_...).
Path Parameters
Body
Response
Pages registered (pending ingestion)
"knowledgebase.crawl"
Same-origin candidates considered before ranking.
New URL documents registered by this crawl.
Selected URLs that already existed in this knowledge base.