AI crawler robots.txt checker
Enter a URL and this tool reads the site’s robots.txt and shows which AI crawlers are allowed or blocked — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and eight more — plus the exact lines to change.
By the CiteSnap team · Published · Last updated
Reads the site's /robots.txt and applies its rules to each AI crawler.
The AI crawlers that matter, and what each one does
Twelve user-agents are worth tracking, and four of them decide whether a major answer engine can cite you at all. CiteSnap weights those four twice as heavily as the rest.
| User-agent | Operator | Blocking it means |
|---|---|---|
GPTBot | OpenAI | No training, and no presence in ChatGPT’s search index |
OAI-SearchBot | OpenAI | No citations in ChatGPT Search results |
ChatGPT-User | OpenAI | ChatGPT cannot fetch your page when a user asks it to |
ClaudeBot | Anthropic | No training and no indexing for Claude |
PerplexityBot | Perplexity | No presence in Perplexity’s index |
Google-Extended | No Gemini training, no AI Overviews — Search itself is unaffected | |
CCBot | Common Crawl | Removal from the corpus many smaller models train on |
Applebot-Extended | Apple | No Apple Intelligence training |
How to allow AI crawlers in robots.txt
Add an explicit Allow: / group for each bot, above any broad Disallow that would otherwise catch it. Most robots.txt parsers apply the most specific matching group, so a named group beats a wildcard.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Three mistakes account for most accidental blocks:
- A blanket
User-agent: * / Disallow: /left over from a staging environment. - A CDN or plugin toggle. Cloudflare’s AI-crawler block, in particular, operates above your robots.txt and will not appear in the file at all — this tool reads what the server serves, so it catches the file-level cases but not a WAF-level block.
- A
Disallowon the path, not the site. Blocking/blog/for a wildcard group blocks it for every AI crawler too. This tool checks the exact path you enter, not just the root.
Blocking is not the only way to disappear
A page can be perfectly crawlable and still never be cited. robots.txt is the first gate, worth 8 of 100 points in the CiteSnap model — the other 92 are about whether the page is parseable, attributable and quotable once a crawler has it.
Run the GEO score checker for the full picture, including whether your content exists before JavaScript runs and whether any section actually answers its own heading.
Frequently asked questions
Is my site blocking GPTBot?
Enter your URL above to find out. The most common cause is not a deliberate block but a plugin or a CDN default: Cloudflare’s "Block AI Scrapers and Crawlers" toggle, Wordfence, and several WordPress SEO plugins add AI crawler Disallow rules without saying so in a way most site owners notice.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for training and for the index ChatGPT searches; OAI-SearchBot is the crawler specifically for ChatGPT Search results. Blocking GPTBot but allowing OAI-SearchBot is a coherent position — no training, yes citation — and CiteSnap treats them as separate bots for exactly that reason.
Does blocking Google-Extended remove me from Google search?
No. Google-Extended controls only whether your content trains Gemini and appears in AI Overviews. Regular Googlebot access, and therefore normal Search ranking, is unaffected. They are separate tokens and separate decisions.
Should I allow every AI crawler?
That depends on what you sell. If your business benefits from being cited as a source — SaaS, documentation, services, media with a subscription funnel — then blocking the crawlers that feed answer engines removes you from an entire discovery channel. If your content is the product and it is sold per-view, blocking training crawlers while allowing search crawlers is a defensible middle position.
Do AI crawlers actually obey robots.txt?
The major ones publish their user-agent tokens and honour robots.txt: OpenAI, Anthropic, Perplexity and Google all document theirs. Compliance is voluntary and unenforceable, and there have been credible reports of less scrupulous crawlers ignoring it. robots.txt is the mechanism for the crawlers you want to influence, not a security control.
How long until a robots.txt change takes effect?
Crawlers cache robots.txt, typically for up to 24 hours. Expect a day or so before a change is reflected, and longer before content crawled under the old rules works its way through an index.