Every AI crawler in 2026: the allow and block table
29 crawlers, verified 2026-07-06. Exact tokens, operators, purpose and robots.txt policy, with sources.
AI crawlers fall into three jobs: training crawlers that feed model weights, search crawlers that index pages to answer questions later, and agents that fetch a page in real time for a user. The table below lists every crawler with a documented user-agent token, its operator, its purpose and whether it respects robots.txt. To block the training bots, use the one-click snippet in the robots.txt checker.
| User-agent token | Operator | Purpose | robots.txt |
|---|---|---|---|
AI2Bot The operator says the token can be used to filter the crawler but makes no explicit compliance claim. Variant: Ai2Bot-Dolma. | Allen Institute for AI | Model training | Not stated |
Amazonbot Checks the host robots.txt or a cache up to 30 days old; crawls as if none exists if it is unreachable. | Amazon | Model training | Respects robots.txt |
ClaudeBot In July 2024 iFixit and Freelancer reported very high request volumes; Anthropic said it respects robots.txt and traffic stopped once a Disallow was added. Reads as a volume story, not confirmed non-compliance. | Anthropic | Model training | Respects robots.txt |
Claude-SearchBot Navigates the web to improve search result quality, not for training, and is independently blockable. | Anthropic | Search index | Respects robots.txt |
Claude-User Fetches a page when a user asks Claude about a URL. Documented as compliant even for these on-demand fetches, unlike some peers. | Anthropic | Live fetch | Respects robots.txt |
Applebot-Extended A permission token that does not itself crawl. It governs reuse of Applebot-fetched data for Apple Intelligence training and never appears in server logs. | Apple | Training permission token | Respects robots.txt |
Bytespider unverified token No first-party ByteDance documentation exists, so the exact token is unverified. Reported in 2023 to 2024 to crawl at very high rates and despite objections, though no compliance policy was ever published to violate. | ByteDance | Model training | Disputed |
CCBot Checks robots.txt first and respects Crawl-delay. Full user agent is CCBot/2.0. Its archive is a common training-data source. | Common Crawl | Model training | Respects robots.txt |
Diffbot Feeds a Knowledge Graph; resold data may feed downstream AI training. Adheres to robots.txt disallow and crawl-delay by default. Variant: Diffbot-User. | Diffbot | Search index | Respects robots.txt |
DuckAssistBot Real-time answer-citation fetch, not training. A robots.txt opt-out takes about 72 hours to propagate. | DuckDuckGo | Live fetch | Respects robots.txt |
Google-Agent Classified by Google as a user-triggered fetcher, which generally ignores robots.txt. Powers agentic browsing actions; added to Google's docs around March 2026. | Live fetch | Ignores robots.txt | |
Google-CloudVertexBot Only fires when a site owner explicitly triggers it to build a Vertex AI agent. | Live fetch | Respects robots.txt | |
Google-Extended A permission token, not a fetcher. Governs reuse of Googlebot-crawled content for Gemini training and grounding; does not affect Search inclusion or ranking. | Training permission token | Respects robots.txt | |
GoogleOther A generic crawler used by various Google teams for one-off research and development. Variants: GoogleOther-Image, GoogleOther-Video. | Model training | Respects robots.txt | |
ImagesiftBot Collects image data. Respects robots.txt including crawl-delay, and falls back to the Googlebot rule if none is specified. | Hive (Castle Global) | Live fetch | Respects robots.txt |
Kagibot Feeds Kagi's paid search index and AI summaries. Falls back to the Googlebot rule if none is specified. Full user agent is Kagibot/1.0. | Kagi | Search index | Respects robots.txt |
meta-externalagent Meta's own casing is lowercase. Documented as compliant, but publisher complaints and Cloudflare's July 2025 finding that Meta AI crawlers generate about 52% of all AI-crawler traffic keep this disputed. | Meta | Model training | Disputed |
meta-externalfetcher Meta's own docs say it may bypass robots.txt because it performs fetches requested by the user. A documented exception. Casing is lowercase. | Meta | Live fetch | Ignores robots.txt |
MistralAI-Index Distinct from MistralAI-User and explicitly excluded from training use. Full user agent is MistralAI-Index/1.0. | Mistral AI | Search index | Respects robots.txt |
MistralAI-User User-triggered fetch, for example in Le Chat. Documented as compliant even for on-demand fetches, and not used for training. | Mistral AI | Live fetch | Respects robots.txt |
ChatGPT-User OpenAI's docs say that because these actions are initiated by a user, robots.txt rules may not apply, and it is not used for automatic crawling. Blocking it is not guaranteed to stop the fetch. | OpenAI | Live fetch | Conditional (user-triggered) |
GPTBot Disallowing GPTBot signals your content should not be used to train foundation models. IP ranges are published at openai.com/gptbot.json. Full user agent is GPTBot/1.3. | OpenAI | Model training | Respects robots.txt |
OAI-AdsBot Validates ChatGPT ad landing pages only. Not used for model training per OpenAI; its robots policy is not explicitly stated. | OpenAI | Live fetch | Not stated |
OAI-SearchBot OpenAI recommends allowing it so your site can appear in ChatGPT search results. Full user agent is OAI-SearchBot/1.3. | OpenAI | Search index | Respects robots.txt |
PerplexityBot Documented as compliant, but Cloudflare found in August 2025 that when blocked, a secondary crawler with a generic browser user agent kept scraping from undeclared IPs, and de-listed Perplexity as a verified bot. Perplexity rebutted; unresolved as of this check. | Perplexity AI | Search index | Disputed |
Perplexity-User Perplexity's own docs say it generally ignores robots.txt for user-triggered fetches. Wired separately found an undisclosed scraper hitting sites that had blocked Perplexity. | Perplexity AI | Live fetch | Ignores robots.txt |
ProRataInc A retrieval answer engine, notable for publisher revenue-share deals. | ProRata (Gist.ai) | Live fetch | Respects robots.txt |
omgili For data licensing. You can opt out via robots.txt. Some third parties report the older omgilibot variant is less compliant; unresolved. | Webz.io | Model training | Respects robots.txt |
YouBot Fully respects robots.txt including user-agent rules and crawl-delay. Full user agent is YouBot/1.0. | You.com | Search index | Respects robots.txt |
Blocking a bot is not always enough
Among the live-fetch bots there are two camps, and the difference is operational. ChatGPT-User, Perplexity-User and meta-externalfetcher are documented by their own operators as potentially bypassing robots.txt, because the fetch is triggered by a person clicking or asking. Claude-User, DuckAssistBot and MistralAI-User are documented as compliant even for those on-demand fetches. So a robots.txt block is expected to work on the second group and is not guaranteed to work on the first.
Credible non-compliance reports
- Perplexity. Cloudflare reported in August 2025 that after PerplexityBot was blocked, a secondary crawler with a generic browser user agent kept fetching from undeclared IPs, and de-listed Perplexity as a verified bot. Perplexity called the analysis inadequate; unresolved as of 2026-07-06.
- ByteDance / Bytespider. Reported in 2023 and 2024 to crawl at very high rates and despite objections. No compliance policy was ever published, and the exact token is unverified against any first-party source.
- Meta. Publisher complaints of crawling despite disallow rules, and Cloudflare found Meta AI crawlers generate about 52% of all AI-crawler traffic, more than Google and OpenAI combined.
How to block the training crawlers
- Decide by category. Blocking training bots costs nothing in search. Blocking a search crawler like OAI-SearchBot can remove you from that answer engine.
- Add a group. List each training token as a User-agent line, then a single Disallow: / for the group. The robots.txt checker generates this snippet for you.
- Do not rely on it for user-triggered fetchers. The bots documented to bypass robots.txt need a different control, such as a firewall or WAF rule, not a Disallow line.
The bigger 2026 shift
Cloudflare has reframed bot management into three behavioural buckets, Search, Agent and Training, arguing the old AI-versus-not-AI split is obsolete now that mainstream search is itself AI-driven. Starting September 15, 2026, new domains onboarding to Cloudflare will have Training and Agent traffic blocked by default on ad-monetized pages, while Search stays allowed. Any block-or-allow decision is now a policy choice, not a technicality.
What is excluded, and why
Some reported crawlers are left out on purpose. cohere-ai, cohere-training-data-crawler and Timpibot have no operator documentation at all, so their exact tokens cannot be confirmed. Microsoft publishes no distinct Copilot crawler; that traffic rides on Bingbot. Bytespider is included but flagged, because it is widely reported even though ByteDance documents nothing.
Common questions
How many AI crawlers are there in 2026?
This list tracks 29 AI crawlers with a documented user-agent token, each verified against its operator's own page where one exists. New ones appear often, and some reported bots are excluded here because no operator documentation confirms them.
Does GPTBot respect robots.txt?
Yes. OpenAI documents GPTBot as respecting robots.txt, and disallowing it opts your content out of foundation-model training. Its separate user-triggered agent, ChatGPT-User, is the one documented as possibly not applying robots.txt.
What is the difference between a training, search and agent crawler?
A training crawler absorbs content into model weights. A search crawler indexes pages to answer questions later. An agent fetches a page in real time on a person's behalf. Blocking a training crawler costs nothing in search, but blocking a search crawler can remove you from that answer engine.
Which AI crawlers ignore robots.txt?
Perplexity-User, meta-externalfetcher and Google-Agent are documented by their operators as potentially bypassing robots.txt because their fetches are user-triggered. Bytespider's compliance is disputed and unverified, and Cloudflare reported a Perplexity stealth crawler continuing after a block.