AI Crawler User-Agents

AI platforms typically operate multiple crawlers for distinct purposes. The user-agent string identifies which system is accessing your site. Whether to allow or block a particular crawler depends on its purpose, not just the company that operates it.

What are the three types of AI crawler?

Training crawlers fetch content to build or update model weights. Blocking them stops your content from being used in future training runs. It has no effect on whether that platform’s AI products currently cite or surface your site.

Search crawlers index content so it can be retrieved and cited when an AI product answers a question, much as Googlebot indexes for Search. Blocking them removes your site from that platform’s AI search results.

Agent crawlers fetch a page live, at the moment a user or an agent asks for it, rather than indexing it in advance. ChatGPT-User, Claude-User and Perplexity-User sit here, and so does Google’s Google-Agent. They behave less like a crawler and more like a browser acting on someone’s behalf, which is the argument their operators use for exempting them from robots.txt, and it is why two of the three are the hardest to control.

The three are separate systems, and the decisions about them are independent. Some operators split them cleanly across user-agents: OpenAI uses GPTBot for training, OAI-SearchBot for search indexing and ChatGPT-User for live fetches. Anthropic mirrors that with ClaudeBot, Claude-SearchBot and Claude-User. Others blur the lines: Perplexity’s PerplexityBot serves both indexing and live retrieval.

This three-way split is no longer just a useful mental model. It is the categorisation Cloudflare uses to gate crawler access at the network layer (see what changes at the CDN below), which means it increasingly determines what actually reaches your server, regardless of what your robots.txt says.

Which AI crawlers are currently active?

User-agent	Purpose
GPTBot (OpenAI)1	Training
OAI-SearchBot (OpenAI)1	ChatGPT Search indexing
ChatGPT-User (OpenAI)1	Live retrieval (user-triggered)
OAI-AdsBot (OpenAI)1	Safety validation of pages submitted as ChatGPT ads
ClaudeBot (Anthropic)2	Training
Claude-SearchBot (Anthropic)2	Search indexing
Claude-User (Anthropic)2	Live retrieval (user-triggered)
PerplexityBot (Perplexity)3	Search indexing and live retrieval
Perplexity-User (Perplexity)3	Live retrieval
Google-Extended (Google)4	Gemini training and Gemini Apps grounding opt-out (a robots.txt token; fetching is done by Google’s existing crawlers)
Google-Agent (Google)5	Agents on Google’s infrastructure browsing and acting (user-triggered)
CCBot (Common Crawl)6	Open training dataset (used by many providers)
Amazonbot (Amazon)7	Training and product improvement
Applebot (Apple)8	Siri and Spotlight indexing
Applebot-Extended (Apple)8	Generative AI features training opt-out
Meta-ExternalAgent (Meta)9	Training foundation models, and indexing content directly
Meta-WebIndexer (Meta)9	Meta AI search indexing
Meta-ExternalFetcher (Meta)9	Live retrieval (user-triggered)
Meta-ExternalAds (Meta)9	Advertising and business products
FacebookExternalHit (Meta)9	Share-preview metadata for links posted to Meta apps
YouBot (You.com)10	Search indexing
ByteSpider (ByteDance)11	Training and search

Note on the crawlers that fit none of the three purposes: OAI-AdsBot and Meta-ExternalAds serve advertising products, and FacebookExternalHit fetches share-preview metadata for links posted to Meta apps. Blocking these has consequences unrelated to AI training or citation.

Note on ChatGPT-User: OpenAI’s documentation now states that its robots.txt tags apply to OAI-SearchBot and GPTBot,¹ having removed ChatGPT-User from that list in a December 2025 revision.¹² Unlike the other crawlers listed here, you cannot rely on robots.txt alone to control it.

Note on Meta’s fetchers: Meta documents that Meta-ExternalFetcher may bypass robots.txt when fulfilling a link fetched at a user’s request, and that FacebookExternalHit may do the same for security and integrity checks.⁹ The other three follow robots.txt normally.

Note on Google-Agent: Google documents it as “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”, and lists it among its user-triggered fetchers, which “generally ignore robots.txt rules” because a user asked for the fetch.⁵ Its user-agent string is a full Chrome string with Google-Agent added, and it comes from the IP ranges in Google’s user-triggered-agents.json, a separate list from the common-crawlers.json ranges Googlebot uses, which is how to tell a genuine visit from a spoofed string.¹³ Google does not say which products send it, so a hit shows that an agent running on Google’s infrastructure fetched the page, not which one.

Note on ByteSpider: ByteDance does not publish IP verification data and ByteSpider has a documented history of robots.txt non-compliance.¹¹ IP-level blocking is more reliable than robots.txt for this crawler.

Note on Microsoft Web IQ: Microsoft’s Web IQ grounding API draws from Bing’s existing index. Jordi Ribas, Microsoft’s President of Search and AI, has said it is used directly in Copilot and by ChatGPT “for some of its web answers”,¹⁴ so it serves part of ChatGPT’s web retrieval, not all of it. No separate Web IQ crawler user-agent exists: BingBot governs what Web IQ can access, and Web IQ inherits Bing’s existing robots.txt compliance. Microsoft says it is “actively engaging with the broader ecosystem through the IETF and other industry forums to help evolve interoperable standards for the AI era”, naming publisher controls and access preferences among what those standards should govern.¹⁵

Do AI crawlers render JavaScript?

Almost none of them do. Vercel and MERJ found that “none of the major AI crawlers currently render JavaScript”, covering GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot; they request JavaScript files without executing them.¹⁶ Gemini and Applebot are the exceptions, both rendering through browser infrastructure.¹⁶ See JavaScript SEO for what that means for how you build a site.

How do you verify a crawler is legitimate?

A user-agent string is self-reported. Any client can claim to be GPTBot or Googlebot. Verification requires checking the source IP, not the string.

Reverse DNS lookup: perform a reverse DNS lookup on the IP that made the request. The resulting hostname should match the crawler’s documented domain (e.g. googlebot.com for Googlebot). Then perform a forward DNS lookup on that hostname and confirm it resolves back to the same IP. This forward-confirmed reverse DNS check is the standard verification method. See Log File Analysis for how to run this check against your server logs.

Published IP lists: several operators provide machine-readable IP ranges:

• OpenAI: published in the bots documentation¹
• Anthropic: https://claude.com/crawling/bots.json²
• Perplexity: https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json³
• Common Crawl: https://index.commoncrawl.org/ccbot.json⁶
• Amazon: published in the Amazonbot documentation⁷

Anthropic’s list carries a creation timestamp of 18 August 2026, making it newer than the others here, so a verification setup built before that date will not be using it and will still be relying on reverse DNS alone.

For crawlers without a published IP list (ByteSpider, YouBot, Meta’s crawlers), log-based pattern analysis and reverse DNS are the available options.

What is replacing reverse-DNS verification?

Reverse DNS and published IP lists are the current standard, but both are workarounds for a missing piece: there is no cryptographic way for a client to prove its identity at the moment of the request. Two separate standardisation efforts are trying to close that gap, and they solve different problems.

Signed agents (bot identity). The Web Bot Auth effort uses HTTP Message Signatures so that an automated client can cryptographically sign its requests, letting a server confirm a crawler is who it claims to be without depending on IP ranges. It is an IETF Internet-Draft, co-authored by Cloudflare and Google, whose stated goal is to let automated HTTP clients sign outbound requests so servers can verify their identity with confidence.¹⁷ This is the direction that would eventually replace reverse-DNS checks for verifying agents like GPTBot or PerplexityBot, and Google says it is already experimenting with it for Google-Agent, under the identity agent.bot.goog.⁵ It is early-stage and not yet widely implemented.

Personhood tokens (human-in-the-loop). In June 2026 Cloudflare, Google, Microsoft, Mozilla and Shopify announced PACT (Private Access Control Tokens), a protocol building on Privacy Pass that lets a site issue anonymous tokens proving a real person, or an agent authorised by one, is behind a request, without tracking the user.¹⁸ PACT does not verify crawler identity and does not replace the user-agent and reverse-DNS methods above; it answers the separate question of whether an automated client is acting on a genuine user’s behalf. See Cloudflare and Browser Makers Announce PACT for detail.

Both are proposed standards in development with no deployment timeline. For now, reverse DNS and published IP lists remain the practical verification methods. The value in tracking these efforts is understanding where access control is heading: from self-reported user-agent strings towards cryptographic proof of both who a client is and on whose behalf it acts.

Which crawlers should you allow or block?

The robots.txt syntax for targeting specific crawlers is covered in Crawlability and robots.txt. The decision of what to do depends on your situation:

No specific concern: the default, where no explicit rules apply to these agents, allows all well-behaved crawlers. Most sites are in this position.

Want AI citations, not training contribution: block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) while leaving the search and agent crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User) untouched. Blocking training crawlers has no effect on search citations, with one exception: Google-Extended also withdraws your content from grounding in Gemini Apps, though not from Google Search or AI Overviews.⁴ If you enforce this at a CDN rather than in robots.txt, check first that the training block does not also catch Googlebot, Bingbot or Applebot.

Paywalled or licensed content: blocking all AI crawlers is a defensible position. Training crawlers extract value without payment; search and agent crawlers may surface excerpts without driving traffic. Be aware that ChatGPT-User may not respect robots.txt: OpenAI’s own wording is that because its fetches are user-initiated, “robots.txt rules may not apply”.¹ A blanket block should still exempt FacebookExternalHit, which fetches share-preview metadata rather than content for training or citation.

ByteSpider: add IP-level blocks via your hosting platform or CDN in addition to robots.txt rules, given the documented non-compliance.

Perplexity-User: robots.txt is not effective. Use server-side access controls (IP blocking or authentication) to prevent Perplexity-User from fetching content. PerplexityBot (the indexing crawler) does respect robots.txt and can be controlled normally.

In August 2025, Cloudflare published research documenting a secondary Perplexity crawler that spoofs a standard Chrome on macOS user-agent string and rotates IP ranges when declared Perplexity crawlers are blocked, generating millions of additional daily requests to affected sites.¹⁹ Cloudflare subsequently de-listed Perplexity as a verified bot and added managed-rule heuristics to detect and block the behaviour. Perplexity disputed the characterisation, stating the activity relates to user-triggered AI Assistant requests rather than automated crawling. Site owners who need to restrict Perplexity access should use CDN-level managed rules rather than relying on IP blocklists alone.

What changes at the CDN

robots.txt asks. A CDN enforces. For a growing share of sites, the decision about whether an AI crawler gets a response is now made before the request reaches the file at your document root.

On 1 July 2026 Cloudflare gave all customers, free accounts included, the ability to sort AI crawlers into the three categories above and manage each separately. It announced that from 15 September 2026 Training and Agent crawlers would be blocked by default on ad-supported pages.²⁰ What shipped on that date is a recommended preset for newly onboarded domains: a site that says it earns from ads gets Training set to Disallow AI Training, which blocks the training-only crawlers run by companies such as OpenAI and Anthropic across the domain, and Agent blocked on pages with ads, while Search stays allowed; a site without ads gets everything allowed.²¹ For those domains the default flips from allow to deny without the site owner editing robots.txt.

The trap: blocking training can block Googlebot. Googlebot, Bingbot and Applebot all fetch for both search indexing and AI training. Cloudflare’s July announcement said a Training block would apply to these multi-purpose crawlers from 15 September,²⁰²² and on that date its Block and Block on pages with ads settings, which “previously did not apply to mixed-use crawlers”, began to, so choosing Block on the Training category stops Googlebot, search included.²¹ The symptom is a crawl collapse with a robots.txt that looks entirely correct, because the block never reaches robots.txt.

The setting that avoids it, added the same day, is “Disallow AI Training”.²¹ It publishes a no-training line in robots.txt and keeps these crawlers fetching for search, because Google, Microsoft and Apple have each stated that opting out of training does not affect search ranking. Bing is the exception for now: Microsoft’s robots.txt support for a no-training preference is targeted for early 2027, and until it launches the setting does not convey the opt-out to Bing. Bing’s own route today is a noarchive robots meta tag, which Bing documents as keeping content out of training for Microsoft’s generative AI models but also as “Do not link in Chat and Copilot”; nocache limits both uses to the URL, title and snippet.²³ Any site that already had Training set to Block or Block on pages with ads, whether through the legacy “Block AI Bots” preset or the granular per-category controls, was migrated to Disallow AI Training automatically. The new-domain preset described above is not retroactive: Cloudflare says existing settings “carry over on their own”. Check your CDN’s Security settings for “Disallow AI Training” specifically, rather than assuming a plain “Block” on the Training category is still the safe choice for these three crawlers.

Before the decisions below, check what your CDN is configured to do rather than assuming your robots.txt is the operative control. See robots.txt and crawlability for the file-level detail.

Frequently asked questions

Does blocking GPTBot affect ChatGPT search results?

No. OAI-SearchBot handles ChatGPT Search indexing. GPTBot is a training crawler only. Blocking it has no effect on whether your site appears in ChatGPT search answers.

Does blocking ClaudeBot affect Claude’s answers?

No. ClaudeBot is Anthropic’s training crawler. Claude-SearchBot handles search indexing, and Claude-User handles live retrieval. Blocking ClaudeBot only affects whether your content is used in future training data.

Can I block one platform’s training crawler but not another’s?

Yes. Each has a distinct user-agent string. You can write separate robots.txt rules for each.

Do all AI crawlers respect robots.txt?

There is no legal requirement to do so. Most major operators comply by policy. ByteSpider is a documented exception.¹¹ ChatGPT-User removed its robots.txt commitment from its documentation in December 2025.¹² Perplexity-User does not respect robots.txt by design; server-side access controls are required to block it.³ Meta documents Meta-ExternalFetcher and FacebookExternalHit as able to bypass it too,⁹ and Google says its user-triggered fetchers, Google-Agent among them, “generally ignore robots.txt rules”.⁵ The pattern is consistent: the crawlers that fetch a page because a user asked for it are the ones exempted from the file.

How do I see which AI crawlers are actually hitting my site?

Server log analysis shows every request your server received. See Log File Analysis. For a free graphical option, Microsoft Clarity’s Bot Activity dashboard reads server-side CDN logs and added a robots.txt violations view in June 2026 that reports which bots are ignoring your directives, broken down by operator, bot name, and the URLs they hit.²⁴ It requires connecting a supported CDN (Fastly, CloudFront, or Cloudflare), so it is not zero-setup, its bot list reflects Microsoft’s own detection rather than an industry standard, and log access charges are billed by your CDN provider.

Footnotes

1. OpenAI bots documentation ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

2. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic Help Center ↩ ↩² ↩³ ↩⁴

3. Perplexity bots documentation ↩ ↩² ↩³ ↩⁴

4. Google-Extended — Google Search Central ↩ ↩²

5. List of Google user-triggered fetchers — Google Crawling Infrastructure, the Google-Agent entry. ↩ ↩² ↩³ ↩⁴

6. Common Crawl CCBot documentation ↩ ↩²

7. Amazonbot — Amazon Developer ↩ ↩²

8. About Applebot — Apple Support ↩ ↩²

9. Meta web crawlers — Meta for Developers ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

10. YouBot documentation — You.com ↩

11. Scrapers selectively respect robots.txt directives — arXiv 2505.21733 (2025) ↩ ↩² ↩³

12. OpenAI revises ChatGPT crawler documentation with significant policy changes — PPC Land ↩ ↩²

13. Verify requests from Google crawlers and fetchers — Google Crawling Infrastructure ↩

14. Microsoft Web IQ (Microsoft names “AI platforms like Copilot and OpenAI”), with Jordi Ribas, President of Search and AI, quoted in Microsoft releases Web IQ, powered by Bing but designed for how AI-agents search — Search Engine Land ↩

15. Announcing Microsoft Web IQ — Bing Search Blog ↩

16. The rise of the AI crawler — Vercel ↩ ↩²

17. HTTP Message Signatures for automated traffic Architecture — IETF Internet-Draft ↩

18. Cloudflare Collaborates With Leading Browsers to Develop a Privacy-First Protocol — Cloudflare ↩

19. Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — Cloudflare Blog ↩

20. Content Independence Day: bringing AI options to every site — Cloudflare ↩ ↩²

21. Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog ↩ ↩² ↩³

22. Cloudflare’s AI crawler rules can block Googlebot — Search Engine Journal ↩

23. Robots meta tags and attributes that Bing supports — Bing Webmaster Tools ↩

24. Spot robots.txt violations in Bot Analytics — Microsoft Clarity Blog ↩