The AI crawler roster: who fetches what, and why
AI traffic is not one thing. GPTBot, ClaudeBot, and CCBot crawl to build training corpora; their visits have no referral value to the site. OAI-SearchBot, PerplexityBot, and Claude-SearchBot index content so answer engines can cite it — those citations can send readers back.
A third class fetches on behalf of a live person: ChatGPT-User or Claude-User retrieving a page because someone asked a question about it right now. Blocking that class does not block a robot in any meaningful sense; it turns away a human who chose an unusual browser.
Same operator, three postures. OpenAI alone runs a training crawler, a search indexer, and a user-fetcher with separate tokens and separate documentation. Any policy that treats 'AI bots' as one category has already made its first mistake.