Crawler policy
This site’s robots.txt sets the following policy for automated crawlers.
Allowed
Section titled “Allowed”- Search engine crawlers (Googlebot, Bingbot, and other standard engines) may index every published page. This documentation exists to be found by people looking for answers.
- AI search crawlers that cite their sources are explicitly allowed: OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), and Claude-SearchBot and Claude-User (Anthropic). When an AI assistant answers a question by searching the web and citing its sources, we want these pages to be citable: accurate, attributed answers about Lead Source help our customers.
Not allowed
Section titled “Not allowed”- Bulk training crawlers are disallowed: GPTBot (OpenAI) and ClaudeBot (Anthropic). Training ingestion copies content without attribution or a route back to the source. That trade isn’t in our customers’ interest, so we opt out.
The distinction we draw
Section titled “The distinction we draw”Crawlers that cite and link the pages they use are welcome; crawlers that absorb content for training without attribution are not. As new crawlers publish their user-agents and purposes, we apply the same test and update robots.txt accordingly.
The published-page list lives in a single sitemap generated from the same source of truth as the pages themselves; pages for unreleased features are excluded from both the site and the sitemap until they ship.