---
title: "Google: Moving to a CDN Can Raise Crawling Before the Cache Is Ready"
description: "Google says a CDN switch can raise Googlebot's crawl rate while the cache is still empty, loading the origin. The fix: pre-warm, allow verified bots, use 429."
tldr: "Moving a site behind a content delivery network (CDN) can raise Googlebot's crawl rate before the CDN's cache can absorb it, Google said on 11 November 2026: fast edge responses lift the crawl capacity limit, while every first request for a URL still reaches the origin server. Google recommends pre-warming the cache, letting verified bots past CDN and firewall challenges, and returning 429 or 503 for a day or two at most if the origin needs relief."
publishDate: 2026-11-12
author: "Liam Hayward"
---
Google published guidance on 11 November 2026 for sites moving behind a content delivery network (CDN), warning that the switch can raise Googlebot's crawl rate at the point the CDN is least able to absorb it. Google's crawl budget documentation already says the crawl capacity limit goes up when a site responds consistently and its response times remain stable or improve. The Search Central blog post, by Cherry Prommawin, a search quality analyst at Google, and Gary Illyes, an analyst on the Google Search team, applies that to a CDN launch: once DNS points at the edge network, "it looks like your server just got a massive hardware upgrade". A new CDN, though, "starts with an empty (cold) cache", and the first request for every URL passes through to the origin. Visitors cluster on a few popular pages that warm up quickly, while crawlers work through the whole URL inventory, and every URL not yet cached is a miss. Google adds that its crawlers do not ramp up immediately and that many CDNs pace connections, which softens the spike without removing it.

## How do you move to a CDN without overloading the origin or blocking Googlebot?

The post gives three steps:

1. **Pre-warm the cache before cutover.** Fetch your highest-traffic pages, upcoming seasonal or promotional pages and most frequently crawled URLs through the CDN "at a controlled pace" before routing all traffic to it. Your server logs show which URLs Googlebot requests most often.
2. **Do not answer the spike with blanket bot protection.** A CDN or web application firewall challenge such as a CAPTCHA stops Google's crawlers, and if it stays in place they back off and pages "will begin to drop out of Search". Use the verified-bot setting most CDNs and firewalls offer, or target specific unwanted bots in robots.txt or firewall rules. Google publishes its crawlers' IP ranges for verification.
3. **Use 429 or 503 as a temporary brake, not 403.** If the origin is still struggling while caches warm, return `429` or `503` for "a few hours, or 1-2 days at most", as Google's guide to reducing crawl rate sets out. Google's post names `403` walls from blanket rules as the response to avoid.

The crawl errors guide covers how each status code affects crawling and indexing, and crawl budget covers how the capacity limit is set.

## Sources

- [Moving to a CDN without overwhelming your origin (or blocking Googlebot), Google Search Central Blog](https://developers.google.com/search/blog/2026/11/crawling-cdn)
- [Crawl budget management, Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawl-budget)
- [Reduce the Google crawl rate, Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate)
- [Verifying Google's crawlers and fetchers, Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [The Non-Canonical Cabinet, episode 5, Google Search Central on YouTube](https://www.youtube.com/watch?v=alGu2A97c98)