Cloudflare's AI Crawler Defaults Arrive, With a New Opt-Out That Protects Googlebot

Cloudflare’s default blocking of Training and Agent crawlers on ad-supported pages, announced on 1 July 2026, took effect as scheduled on 15 September 2026. Alongside it, Cloudflare shipped the fix for the risk that rollout created: a new “Disallow AI Training” setting for crawlers that serve more than one purpose, such as Googlebot, Bingbot and Applebot. Applying it blocks training use while leaving search indexing untouched, because Google, Microsoft and Apple have each committed that opting out of training does not affect search ranking. Any site that already had Training set to Block or Block on pages with ads, whether through the legacy “Block AI Bots” preset or the granular per-category controls, was migrated to the new setting automatically, no action needed. The stricter 15 September default is separate and not retroactive: it applies only to domains onboarding from that date, so an existing site’s configuration carries over unchanged either way. Cloudflare also introduced Bot Preference Sync, live since 21 August 2026, which keeps a site’s published robots.txt automatically matched to its dashboard settings so the two cannot drift apart.

If you are behind Cloudflare, check your Security settings for “Disallow AI Training” specifically, rather than assuming a plain “Block” on the Training category is still the safe choice.

Sources

• Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog
• Say it once: introducing Bot Preference Sync — Cloudflare Blog