Industry Snippet

Google: Moving to a CDN Can Raise Crawling Before the Cache Is Ready

Markdown
View as Markdown Open this page as plain text
This page as Markdown, ready to paste into ChatGPT, Claude or a doc.
RSS
Flat cartoon illustration of a round character in Google's brand colours, wearing a propeller cap, sitting on a purple bench in an old-fashioned train compartment with its arms flung wide and a speech bubble reading WE'LL TAKE THE LOT. Through the open door, a tiered refreshment trolley is piled with page-shaped sweets, in jars labelled OLD POSTS, TAG PAGES and PAGE 47.
Crawlers request every URL on a site, including the long tail visitors rarely open, which is why Google advises pre-warming a new CDN's cache before switching traffic to it. Illustration: AI-generated.

Google published guidance on 11 November 2026 for sites moving behind a content delivery network (CDN), warning that the switch can raise Googlebot’s crawl rate at the point the CDN is least able to absorb it. Google’s crawl budget documentation already says the crawl capacity limit goes up when a site responds consistently and its response times remain stable or improve. The Search Central blog post, by Cherry Prommawin, a search quality analyst at Google, and Gary Illyes, an analyst on the Google Search team, applies that to a CDN launch: once DNS points at the edge network, “it looks like your server just got a massive hardware upgrade”. A new CDN, though, “starts with an empty (cold) cache”, and the first request for every URL passes through to the origin. Visitors cluster on a few popular pages that warm up quickly, while crawlers work through the whole URL inventory, and every URL not yet cached is a miss. Google adds that its crawlers do not ramp up immediately and that many CDNs pace connections, which softens the spike without removing it.

How do you move to a CDN without overloading the origin or blocking Googlebot?

The post gives three steps:

  1. Pre-warm the cache before cutover. Fetch your highest-traffic pages, upcoming seasonal or promotional pages and most frequently crawled URLs through the CDN “at a controlled pace” before routing all traffic to it. Your server logs show which URLs Googlebot requests most often.
  2. Do not answer the spike with blanket bot protection. A CDN or web application firewall challenge such as a CAPTCHA stops Google’s crawlers, and if it stays in place they back off and pages “will begin to drop out of Search”. Use the verified-bot setting most CDNs and firewalls offer, or target specific unwanted bots in robots.txt or firewall rules. Google publishes its crawlers’ IP ranges for verification.
  3. Use 429 or 503 as a temporary brake, not 403. If the origin is still struggling while caches warm, return 429 or 503 for “a few hours, or 1-2 days at most”, as Google’s guide to reducing crawl rate sets out. Google’s post names 403 walls from blanket rules as the response to avoid.

The crawl errors guide covers how each status code affects crawling and indexing, and crawl budget covers how the capacity limit is set.

Sources

More news

Add as a preferred source on Google