Cloudflare Splits AI Crawlers Into Search, Agent and Training, With Default Blocks From 15 September

Cloudflare has changed how it manages automated traffic for every site on its network, including those on free plans. Announced on 1 July 2026, the update sorts crawlers into three categories by what they actually do, Search, Agent and Training, and lets site owners allow or block each one independently. Two default changes then take effect on 15 September that site owners relying on organic search should understand before those defaults arrive.

The reason this matters for SEO is not the AI-blocking headline. It is a quieter mechanic buried in how Cloudflare handles crawlers that do more than one job.

What are Cloudflare’s three crawler categories?

Cloudflare now groups crawlers by behaviour rather than by whether they count as “AI”:

• Search indexes a site so it can answer questions and send referral traffic later. This is the traditional search-crawler role.
• Agent covers real-time bots acting on behalf of a person, such as a chat assistant’s fetch bot or a browser-use agent like Gemini or Claude operating inside Chrome.
• Training covers crawling that pulls content to train or fine-tune a model, with no direct referral in return.

Every customer, including free accounts, can now see these categories and set a separate rule for each. That granularity is the useful part of the announcement: you can keep search crawlers welcome while restricting training access, rather than making one blunt allow-or-block decision.

What changes on 15 September?

Two defaults change for new sites and new customers. On pages that display ads, Training and Agent crawlers will be blocked by default, while Search crawlers remain allowed. Existing configurations are not rewritten, but anyone setting up a new site should expect the stricter starting point.

Cloudflare is also extending its pay-per-crawl approach, which lets publishers charge AI companies for access rather than only allowing or blocking them. That mechanism is aimed at the training economy rather than at search indexing.

Why could this block Googlebot?

This is the part with direct organic-search consequences. Many crawlers are multi-purpose: Google, Microsoft and Apple use Googlebot, Bingbot and Applebot to crawl both for search results and to gather data that feeds AI features. Cloudflare says it treats a multi-purpose crawler by the strictest rule that applies to it. So if a site blocks the Training category, a combined crawler that performs both Search and Training can be blocked outright, taking your search indexing down with it.

In practice, a site owner who ticks “block AI training” to keep models out could unintentionally stop Googlebot or Bingbot from crawling for ordinary search indexing. That is a self-inflicted deindexing risk dressed up as an AI-privacy setting, and it is exactly the sort of change worth auditing before the September defaults arrive.

What should site owners do?

If your site sits behind Cloudflare, this is a review-your-settings moment rather than a cause for alarm. Before 15 September:

• Check which crawler categories you currently allow, and confirm Search is not being caught by a broad Training block.
• If your priority is organic visibility, keep the Search category allowed for Google, Microsoft and Apple, and make training decisions separately.
• Treat Agent access as its own question. Whether you want browser agents reaching your pages depends on whether your site expects agents to act on it, which matters more for commerce and booking than for editorial content.

None of these controls change how Google ranks your pages. They change whether Google’s crawler can reach them at all, which is a more fundamental problem if it goes wrong. As always with a network-level setting like this, confirm the specifics against Cloudflare’s own documentation before changing your configuration.

This sits alongside a related reminder from Google the same week: John Mueller said self-declared files such as Content Signals and llms.txt have no confirmed crawler effect. The Cloudflare controls are different in kind, because they are enforced at the network edge and do take effect, which is precisely why a misconfiguration here bites where a robots.txt preference file would not.

Sources

• Your site, your rules: new AI traffic options for all customers — Cloudflare Blog
• Cloudflare’s AI Crawler Rules Can Block Googlebot — Search Engine Journal
• Cloudflare’s new policy pushes AI companies to pay for publishers’ content — TechCrunch
• Cloudflare Sets September 15 Deadline for AI Companies to Separate Training Crawlers — MLQ News