Tool 01
Which AI bots can read your site?
Type your domain. We fetch your robots.txt and go through it bot by bot: ChatGPT, Claude, Perplexity, Gemini and 17 others. If something is blocked, you get the exact lines to change.
AI search visibility
of AI search & user bots can reach your homepage
Enter your domain above. We fetch your robots.txt and check it rule by rule.
We check the 21 AI crawlers below, grouped into search, user-triggered and training bots.
We also look for a Sitemap: line and syntax problems.
| Bot | What it does | Status |
|---|---|---|
OAI-SearchBot OpenAI · AI search | Indexes pages for ChatGPT search results and citations. Allow. Blocking it keeps you out of ChatGPT search answers. | Not checked
|
ChatGPT-User OpenAI · User-triggered | Fetches a page live when a ChatGPT user (or custom GPT) asks about it. Allow. It only visits when a real user requests your content. | Not checked
|
GPTBot OpenAI · Model training | Crawls content that may be used to train OpenAI models. Your choice. Blocking does not remove you from ChatGPT search (that is OAI-SearchBot). | Not checked
|
ClaudeBot Anthropic · Model training | Collects public web content for training Anthropic's Claude models. Your choice. Allowing helps Claude know your brand. | Not checked
|
Claude-SearchBot Anthropic · AI search | Indexes content to improve Claude's search results. Allow if you want to be cited in Claude answers. | Not checked
|
Claude-User Anthropic · User-triggered | Fetches pages when a Claude user asks about them. Allow. User-initiated visits. | Not checked
|
anthropic-ai Anthropic · Model training | Legacy Anthropic token still honoured in many robots.txt files. Your choice; keep it consistent with ClaudeBot. | Not checked
|
PerplexityBot Perplexity · AI search | Indexes pages so Perplexity can surface and cite them. Allow. Blocking keeps you out of Perplexity answers. | Not checked
|
Perplexity-User Perplexity · User-triggered | Fetches pages live to answer a user's question. Allow. User-initiated visits. | Not checked
|
Google-Extended Google · Model training | Controls whether Gemini models may train on/ground with your content. It does not affect Google Search or AI Overviews ranking. Your choice. AI Overviews are governed by Googlebot, not this token. | Not checked
|
Googlebot Google · AI search | Google Search crawler, which also powers AI Overviews and AI Mode. Must allow. Blocking it removes you from Google entirely. | Not checked
|
Bingbot Microsoft · AI search | Bing crawler; Bing's index powers Copilot and feeds several AI assistants. Allow. | Not checked
|
Applebot-Extended Apple · Model training | Controls whether Apple may use content crawled by Applebot to train Apple Intelligence. Your choice. Siri/Spotlight use Applebot, not this token. | Not checked
|
Applebot Apple · AI search | Powers Siri and Spotlight suggestions. Allow. | Not checked
|
CCBot Common Crawl · Model training | Builds the open Common Crawl dataset used to train many LLMs. Your choice. Many models learn about brands from Common Crawl. | Not checked
|
Bytespider ByteDance · Model training | Collects training data for ByteDance models (e.g. Doubao). Commonly blocked; aggressive crawl rate. | Not checked
|
meta-externalagent Meta · Model training | Crawls for Meta AI model training and products. Your choice. | Not checked
|
Amazonbot Amazon · AI search | Used by Alexa and Amazon AI features to answer questions. Allow if you want Alexa answers to cite you. | Not checked
|
DuckAssistBot DuckDuckGo · AI search | Fetches pages for DuckDuckGo's AI-assisted answers. Allow. | Not checked
|
MistralAI-User Mistral · User-triggered | Fetches pages when Le Chat users request them. Allow. | Not checked
|
cohere-ai Cohere · Model training | Cohere's crawler for its AI products. Your choice. | Not checked
|
Why AI crawler access matters
AI assistants can only recommend businesses they can read. Before ChatGPT, Perplexity or Claude can cite your page, their crawlers have to be allowed to fetch it. One overly broad line in robots.txt, often added by a plugin or a security setting, can quietly remove you from AI answers.
Three kinds of AI bots
- AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) build the indexes AI answers are drawn from. Blocking these costs you citations and referral traffic.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit only when a person asks the assistant about your page. Blocking them breaks those answers.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider) collect data to train models. Blocking them is a legitimate business choice and doesn't directly remove you from AI search.
How to read your results
Allowed means the bot can fetch your homepage and no path rules apply. Partly restricted means the homepage is open but some paths are disallowed; that's normal for admin or cart pages. Blocked means the bot is disallowed from your homepage, usually by Disallow: /.
We follow the official robots.txt standard (RFC 9309): a bot obeys the group that names it, and falls back to User-agent: * only if no group names it. The most specific matching rule wins.
A recommended AI-friendly robots.txt
# Allow AI search and user-triggered bots
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://yourdomain.com/sitemap.xmlNext step: help AI understand your site with an llms.txt file.
FAQ
Questions
Rather not do this yourself? We'll do the whole thing and send you the receipts.