--- title: "Cloudflare Splits AI Crawlers Into Search, Agent and Training, With Default Blocks From 15 September" description: "Cloudflare now sorts every crawler into Search, Agent or Training and, from 15 September, blocks Training and Agent bots by default on ad-supported pages." tldr: "On 1 July 2026 Cloudflare gave all customers, including free accounts, the ability to sort AI crawlers into three categories, Search, Agent and Training, and manage each separately. From 15 September, Training and Agent crawlers will be blocked by default on ad-supported pages for new sites and new customers, while Search crawlers stay allowed. Because Cloudflare applies the strictest matching rule to multi-purpose crawlers, a site that blocks Training can also block combined bots such as Googlebot, Bingbot and Applebot, which crawl for both search indexing and AI training." publishDate: 2026-07-09 updatedDate: 2026-09-15 author: "Liam Hayward" --- Cloudflare has changed how it manages automated traffic for every site on its network, including those on free plans. Announced on 1 July 2026, the update sorts crawlers into three categories by what they actually do, Search, Agent and Training, and lets site owners allow or block each one independently. Two default changes then take effect on 15 September that site owners relying on organic search should understand before those defaults arrive. The reason this matters for SEO is not the AI-blocking headline. It is a quieter mechanic buried in how Cloudflare handles crawlers that do more than one job. ## What are Cloudflare's three crawler categories? Cloudflare now groups crawlers by behaviour rather than by whether they count as "AI": - **Search** indexes a site so it can answer questions and send referral traffic later. This is the traditional search-crawler role. - **Agent** covers real-time bots acting on behalf of a person, such as a chat assistant's fetch bot or a browser-use agent like Gemini or Claude operating inside Chrome. - **Training** covers crawling that pulls content to train or fine-tune a model, with no direct referral in return. Every customer, including free accounts, can now see these categories and set a separate rule for each. That granularity is the useful part of the announcement: you can keep search crawlers welcome while restricting training access, rather than making one blunt allow-or-block decision. ## What changes on 15 September? Two defaults change for new sites and new customers. On pages that display ads, Training and Agent crawlers will be blocked by default, while Search crawlers remain allowed. Existing configurations are not rewritten, but anyone setting up a new site should expect the stricter starting point. Cloudflare is also extending its pay-per-crawl approach, which lets publishers charge AI companies for access rather than only allowing or blocking them. That mechanism is aimed at the training economy rather than at search indexing. ## Why could this block Googlebot? This is the part with direct organic-search consequences. Many crawlers are multi-purpose: Google, Microsoft and Apple use Googlebot, Bingbot and Applebot to crawl both for search results and to gather data that feeds AI features. Cloudflare says it treats a multi-purpose crawler by the strictest rule that applies to it. So if a site blocks the Training category, a combined crawler that performs both Search and Training can be blocked outright, taking your search indexing down with it. In practice, a site owner who ticks "block AI training" to keep models out could unintentionally stop Googlebot or Bingbot from crawling for ordinary search indexing. That is a self-inflicted deindexing risk dressed up as an AI-privacy setting, and it is exactly the sort of change worth auditing before the September defaults arrive. ## What should site owners do? If your site sits behind Cloudflare, this is a review-your-settings moment rather than a cause for alarm. Before 15 September: - Check which crawler categories you currently allow, and confirm Search is not being caught by a broad Training block. - If your priority is organic visibility, keep the Search category allowed for Google, Microsoft and Apple, and make training decisions separately. - Treat Agent access as its own question. Whether you want browser agents reaching your pages depends on whether your site expects agents to act on it, which matters more for commerce and booking than for editorial content. None of these controls change how Google ranks your pages. They change whether Google's crawler can reach them at all, which is a more fundamental problem if it goes wrong. As always with a network-level setting like this, confirm the specifics against Cloudflare's own documentation before changing your configuration. This sits alongside a related reminder from Google the same week: John Mueller [said self-declared files such as Content Signals and llms.txt have no confirmed crawler effect](/seo-news-updates/google-content-signals-llms-txt-no-effect/). The Cloudflare controls are different in kind, because they are enforced at the network edge and do take effect, which is precisely why a misconfiguration here bites where a robots.txt preference file would not. ## Update: a training opt-out that keeps Googlebot indexing (September 2026) On 15 September 2026, the date the stricter defaults above took effect, Cloudflare also shipped the fix for the Googlebot risk this article warned about. Googlebot, Bingbot and Applebot now carry an "Accountable" designation, and site owners can apply a new "Disallow AI Training" setting to them instead of a flat block. That setting refuses training use while leaving search indexing untouched, because Google, Microsoft and Apple have each committed that opting out of training does not affect search ranking. Any site that already had Training set to Block or Block on pages with ads, whether through the legacy "Block AI Bots" preset or the granular per-category controls, was migrated to this new setting automatically, no action needed. The stricter 15 September default described above is separate and not retroactive: Cloudflare applies it only to domains onboarding from that date, so an existing site's configuration carries over unchanged either way. Separately, Cloudflare's Bot Preference Sync, live since 21 August 2026, now keeps a site's published robots.txt automatically matched to whatever is set in the dashboard, so the two cannot drift apart and give a crawler grounds to disregard the stated preference. The practical result: a site that wants to keep training crawlers out no longer has to choose between that and staying in Google's index. Anyone who audited settings in July should check for "Disallow AI Training" specifically, rather than assuming a plain "Block" is still the safe option for Training. ## Sources - [Your site, your rules: new AI traffic options for all customers — Cloudflare Blog](https://blog.cloudflare.com/content-independence-day-ai-options/) - [Cloudflare's AI Crawler Rules Can Block Googlebot — Search Engine Journal](https://www.searchenginejournal.com/cloudflares-ai-crawler-rules-can-block-googlebot/581385/) - [Cloudflare's new policy pushes AI companies to pay for publishers' content — TechCrunch](https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/) - [Cloudflare Sets September 15 Deadline for AI Companies to Separate Training Crawlers — MLQ News](https://mlq.ai/news/cloudflare-sets-september-15-deadline-for-ai-companies-to-separate-training-crawlers-or-face-default-blocks/) - [Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) - [Say it once: introducing Bot Preference Sync — Cloudflare Blog](https://blog.cloudflare.com/bot-preference-sync/)