Websites
Set up Websites
Ingest public web pages — a handbook site, a help centre, published policies.
| URLs | One page URL per line. |
|---|---|
| Sitemap URL | A sitemap.xml — every page listed in it is ingested. Sitemap index files (the WordPress/Shopify layout) and gzipped sitemaps work too. Combine with the URL list or use either alone. |
Pages behind a login can't be reached. Each sync re-checks pages and updates any that changed. Only the URLs you configure are fetched — links are not crawled and robots.txt is not consulted, so point it exactly at what you want indexed; requests are paced so your site is never hammered.
Was this page helpful?
Last updated 29 Aug 2026