GPTBot vs OAI-SearchBot: which AI crawlers to block, and which you can't afford to

A lot of sites added User-agent: GPTBot / Disallow: / in 2023, when the worry was AI companies training on their content. Reasonable. The problem is that "AI bots" now covers three very different jobs, and blocking the wrong one quietly removes you from the answers your customers read.

Three kinds of bots

Search indexers build the index that AI answers are drawn from: OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, and of course Googlebot and Bingbot. Block these and you can't be cited.

User-triggered fetchers visit only when a person asks the assistant to look at a page: ChatGPT-User, Claude-User, Perplexity-User. Blocking them breaks the moment a customer pastes your link into a chat.

Training crawlers collect data to train future models: GPTBot, ClaudeBot, CCBot (Common Crawl), Google-Extended, Applebot-Extended, Bytespider. Blocking these is a legitimate choice. It doesn't remove you from today's AI search.

The common mistakes

• Blocking Google-Extended to "stay out of AI Overviews". It doesn't work that way. AI Overviews are part of Google Search and use Googlebot. Google-Extended only governs Gemini training and grounding.
• A blanket User-agent: * / Disallow: / left over from a staging site. It blocks everything that doesn't have its own group.
• Assuming robots.txt is the whole story. CDN bot-protection settings can block AI crawlers before they ever read your robots.txt.

A sensible default

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Optional: opt out of model training
User-agent: GPTBot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

Not sure what yours says right now? Run the AI Crawler Checker. It reads your live robots.txt and lists every bot by name.