Cloudflare's AI Crawler Defaults Arrive, With a New Opt-Out That Protects Googlebot

Cloudflare’s plan to block Training and Agent crawlers by default on ad-supported pages, announced on 1 July 2026, arrived on 15 September 2026 as a recommended preset for newly onboarding domains. A site that says it earns from ads is offered Training set to a new “Disallow AI Training” setting and Agent blocked on pages with ads, with Search allowed; a site without ads is offered everything allowed, and either can be changed during onboarding. The same day, Cloudflare’s Block and Block on pages with ads settings, which previously did not apply to crawlers serving more than one purpose, began to apply to them, so choosing Block on the Training category now stops Googlebot, Bingbot and Applebot, search included. Disallow AI Training is the setting that avoids that: it publishes a no-training line in robots.txt and keeps those crawlers fetching for search, because Google, Microsoft and Apple have each stated that opting out of training does not affect search ranking. Bing is the exception for now: Microsoft’s robots.txt support for a no-training preference is targeted for early 2027, and until then the setting does not convey the opt-out to Bing. Any site that already had Training set to Block or Block on pages with ads, whether through the legacy “Block AI Bots” preset or the granular per-category controls, was migrated to Disallow AI Training automatically, and Cloudflare says existing settings “carry over on their own”. Cloudflare also introduced Bot Preference Sync, live since 21 August 2026, which keeps a site’s published robots.txt automatically matched to its dashboard settings so the two cannot drift apart.

Update, 27 September 2026: This article first said the July defaults took effect as scheduled and that Disallow AI Training fixed a risk the rollout had created. Cloudflare’s 15 September post says Block “previously did not apply to mixed-use crawlers”, so a Training block could not catch Googlebot before that date, and what shipped for new domains is the onboarding preset described above. The Bing exception has also been added.

If you are behind Cloudflare, check your Security settings for “Disallow AI Training” specifically, rather than assuming a plain “Block” on the Training category is still the safe choice.

Sources

• Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog
• Say it once: introducing Bot Preference Sync — Cloudflare Blog