robots.txt AI crawler checker

Paste your robots.txt to see which AI crawlers it allows or blocks, with the deciding rule for each. The verdicts use the same longest-match precedence Google applies. Everything runs in your browser.

Allow or block verdict per AI crawler for the pasted rules.
Crawler Operator Type Verdict Deciding rule
AI2Bot Allen Institute for AI Training allowed default
Amazonbot Amazon Training allowed default
ClaudeBot Anthropic Training allowed default
Claude-SearchBot Anthropic Search allowed default
Claude-User Anthropic Agent allowed default
Applebot-Extended Apple Training allowed default
Bytespider unverified ByteDance Training allowed default
CCBot Common Crawl Training allowed default
Diffbot Diffbot Search allowed default
DuckAssistBot DuckDuckGo Agent allowed default
Google-Agent Google Agent allowed default
Google-CloudVertexBot Google Agent allowed default
Google-Extended Google Training allowed default
GoogleOther Google Training allowed default
ImagesiftBot Hive (Castle Global) Agent allowed default
Kagibot Kagi Search allowed default
meta-externalagent Meta Training allowed default
meta-externalfetcher Meta Agent allowed default
MistralAI-Index Mistral AI Search allowed default
MistralAI-User Mistral AI Agent allowed default
ChatGPT-User OpenAI Agent allowed default
GPTBot OpenAI Training allowed default
OAI-AdsBot OpenAI Agent allowed default
OAI-SearchBot OpenAI Search allowed default
PerplexityBot Perplexity AI Search allowed default
Perplexity-User Perplexity AI Agent allowed default
ProRataInc ProRata (Gist.ai) Agent allowed default
omgili Webz.io Training allowed default
YouBot You.com Search allowed default
Block the training crawlers

A ready-made group that opts out of the model-training bots. User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt, so this cannot stop them. 

 
 
 
Parsed in your browser. Your robots.txt is never sent anywhere.

 
   
 
 

This tool reads your robots.txt and, for every known AI crawler, tells
          you whether it is allowed or blocked for a path, following Google's
          rules: the most specific user-agent group wins, and within it the
          longest matching rule wins, with Allow breaking ties. The crawler list
          is maintained in one place and last verified 2026-07-06.

 
 
Blocking a crawler is not always enough
 

The important honesty point: among live-fetch bots there are two camps.
        ChatGPT-User, Perplexity-User and meta-externalfetcher are documented by
        their own operators as potentially bypassing robots.txt because the fetch
        is triggered by a person. Claude-User, DuckAssistBot and MistralAI-User
        are documented as compliant even for those on-demand fetches. Blocking the
        first camp is not guaranteed to stop the fetch; blocking the second is.
        The full breakdown, with sources, is on the
complete AI crawler list.

 
Training, search and agent crawlers are different decisions
 

Blocking a training crawler removes you from a future model's training
        data but costs you nothing in search. Blocking a search-index crawler like
        OAI-SearchBot or PerplexityBot can remove you from that answer engine's
        results, which is usually the opposite of what you want. Decide per
        category, not with one blanket rule.

 
   
 
Common questions
 
 
 How do I block AI crawlers in robots.txt? 
Add a group that lists the training crawlers as User-agent lines followed by Disallow: /. The snippet button on this page generates one for you. Remember that user-triggered fetchers such as ChatGPT-User and Perplexity-User document that they may ignore robots.txt.
 
 Does GPTBot respect robots.txt? 
Yes. OpenAI documents GPTBot as respecting robots.txt, and disallowing it signals that your content should not be used to train foundation models. OpenAI's user-triggered ChatGPT-User is the one that may not apply robots.txt rules.
 
 Will blocking AI crawlers hurt my Google ranking? 
No. Blocking training tokens like GPTBot, CCBot and Google-Extended does not affect Googlebot or Bingbot, which handle search ranking separately. Blocking search-index AI bots such as OAI-SearchBot or PerplexityBot can, however, remove you from those answer engines.
 
 Do all AI crawlers obey robots.txt? 
No. Some are documented to bypass it, including Perplexity-User, meta-externalfetcher and Google-Agent, because their fetches are user-triggered. Others, such as Bytespider, have disputed or unverified compliance. The table flags each one.