--- title: "Crawl Errors" description: "What crawl errors are, how to find them in Search Console's Pages report, and how to tell a genuine problem from a normal, intentional exclusion." tldr: "A crawl error is any failure that stops Google fetching a URL it tried to reach: server errors (5xx), DNS and connectivity failures, redirect problems, and blocked resources. Google retired the old Crawl Errors report in 2019, so crawl and indexing problems now live in Search Console's Pages report. The critical skill is triage: many 'errors' are intentional and correct (noindex pages, canonicalised duplicates, disallowed URLs), while server errors, soft 404s and 'discovered but not crawled' at scale signal real trouble. Fix the genuine failures, confirm the rest are deliberate, and do not chase every non-indexed URL to zero." publishDate: 2026-08-03 updatedDate: 2026-11-04 author: "Liam Hayward" --- A crawl error is what happens when Google tries to fetch one of your URLs and cannot. The fetch may fail because your server returned an error, the domain would not resolve, a redirect looped, or a directive blocked access. Crawl errors matter because a page Google cannot crawl cannot be indexed, and Google says that when a site starts returning server errors (5xx status codes) or rate-limiting responses such as 429, it lowers the crawl limit and crawls less.[^8] The skill is less in finding them than in telling a real failure from a normal, intentional exclusion. This is the diagnostic layer beneath [indexing](/technical-seo/indexing/): indexing is whether a page makes it into Google's index, and crawl errors are one of the reasons a page never gets the chance. ## Where do you find crawl errors now? Google retired the standalone Crawl Errors report in 2019.[^1] Crawl and indexing problems now live in the **Pages** report (formerly Index Coverage) inside [Google Search Console](/technical-seo/google-search-console/). It groups URLs into indexed and "not indexed", and lists the reason for each non-indexed group, along with warnings for pages that are indexed despite an issue. One thing the Pages report does not surface as a reason code is a DNS or connectivity failure. Those show up through URL Inspection and the Crawl Stats report instead, so a site that cannot be reached at all will not necessarily announce itself in the place you are looking. The **URL Inspection** tool complements the report: for any single URL it shows the last crawl outcome, whether the page is indexed, and what Google encountered, which is how you diagnose a specific page rather than a category. ## What are the main types of crawl error? **Server errors (5xx).** Your server returned a 500, 502, 503, or similar when Google requested the page. These are genuine problems: they block crawling and, at scale, signal an unreliable site that Google will crawl more cautiously. Persistent 5xx errors are a priority fix. **DNS and connectivity failures.** Google could not resolve your domain or reach the server at all. Usually intermittent hosting or DNS issues; if sustained, they are serious because they can affect the whole site. **Redirect errors.** Redirect chains that are too long, loops, or redirects to broken URLs. These waste crawl budget and can strand pages. Resolve with clean, single-hop [redirects](/technical-seo/redirects/). **404 (not found).** A requested URL does not exist. A 404 is not inherently a problem, removed pages should 404 or 410, but a 404 that receives internal links or external links is a leak worth fixing. **Soft 404.** The page returns a 200 status but has no meaningful content, or tells the user the content is missing. Google treats it as a not-found in disguise. These need a real fix: either restore content, return a proper 404, or redirect to a relevant page. **Blocked URLs.** The URL is disallowed in [robots.txt](/technical-seo/crawlability-and-robots-txt/). Whether this is an error depends entirely on intent, which the section below on telling a real error from an intentional exclusion covers. A `noindex` page is a different case and not a crawl error at all: it was fetched successfully, and the exclusion is an indexing decision rather than a fetch failure. ### Which status codes actually slow Googlebot down? This is where intuition misleads people. Only two kinds of status code reduce crawl rate: `5xx` server errors, and `429`. Google is explicit that its crawlers "treat the `429` status code as a signal that the server is overloaded, and it's considered a server error", and that `5xx` and `429` together "prompt Google's crawlers to temporarily slow down with crawling".[^3] Failures with no status code count too: Google treats network timeouts, connection resets and DNS errors "similarly to `5xx` server errors", crawling "immediately starts slowing down", and indexed URLs that stay unreachable "will be removed from Google's index within days".[^6] Everything else in the 4xx family does not. In Google's words, "the `4xx` status codes, except `429`, have no effect on crawl rate", and it warns directly against the common workaround: "Don't use `401` and `403` status codes for limiting the crawl rate."[^3] All 4xx except 429 are treated identically, signalling to the indexing pipeline that the content does not exist, which removes the URL if it was previously indexed. A wall of 403s from a bot-mitigation rule will not throttle Googlebot at all; it will deindex the affected URLs instead, which is worse, and it will do so without any signal that reads as a crawl-rate problem. If you need to shed crawl load during an incident, `503` or `429` is the correct response, because both preserve indexed URLs while you recover. That is a short-term measure: Google advises against serving them for "longer than 1-2 days", because a URL that returns them for multiple days "may be dropped from Google's index".[^4] ## How do you tell a real error from an intentional exclusion? Most of the "not indexed" volume in a healthy site's Pages report is deliberate and correct. Treating every non-indexed URL as a problem is a common way to waste time here. Work through the reasons and classify each: The reason names below are the labels Google uses in the report today. They have been renamed more than once, so older guides quote strings you will not find in the interface.[^2] **Almost always intentional and fine:** - **URL marked 'noindex':** you asked for this on pages you do not want indexed (thank-you pages, filtered views). - **Alternate page with proper canonical tag:** Google chose your declared canonical; the duplicate is correctly folded in. - **Blocked by robots.txt:** expected for admin, cart, or parameter URLs you deliberately disallowed. - **Page with redirect:** the old URL redirects; it should not be indexed. **Investigate, often a real problem:** - **Server error (5xx):** fix these. - **Soft 404:** fix content, status, or redirect. - **Blocked due to unauthorized request (401)**, **Blocked due to access forbidden (403)** and **URL blocked due to other 4xx issue:** usually a bot-mitigation rule, WAF or rate limiter treating Googlebot as hostile traffic rather than anything you configured deliberately. Google is direct about the 403 case: "Googlebot never provides credentials, so your server is returning this error incorrectly."[^2] - **Discovered - currently not indexed** at scale: Google knows the URL but has not crawled it. Google's stated cause is capacity, not quality: "Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl."[^2] Thin content and weak internal linking are plausible secondary readings, but check [crawl budget](/technical-seo/crawl-budget/) and server response first. - **Crawled - currently not indexed** at scale: Google has crawled the page, so it has seen the content, but has not added it to the index. Google calls this "a common message for many sites" that "doesn't necessarily indicate an issue that needs attention", because Google doesn't index every page on the web.[^2] For pages that matter, use URL Inspection to rule out a block; if many sit here with no technical reason, check them against Google's guidance on helpful content.[^7] - **Duplicate without user-selected canonical** and **Duplicate, Google chose different canonical than user:** worth checking your canonical signals are consistent. The rule of thumb: a handful of a given reason is usually noise; a large or growing count of a problem reason is a signal. Fix the genuine failures, confirm the exclusions are deliberate, and resist driving every non-indexed number to zero, which is neither achievable nor desirable. ### What about crawl errors for AI crawlers? Search Console reports on Googlebot. Nothing in it describes what [AI crawlers](/ai-search/ai-crawlers/) received, so the 403 problem above has a silent twin: a bot-mitigation rule or WAF that blocks GPTBot, ClaudeBot or PerplexityBot produces no error anywhere in Search Console, because Google was never the crawler being turned away. It is visible only in your own logs: those of the CDN or WAF that applied the block, which may not be your server's if the request never reached it. This is worth a deliberate check rather than an assumption, because a bot-mitigation setting can block AI crawlers without anyone editing robots.txt: since 15 September 2026, Cloudflare has offered new domains that serve ads a recommended configuration that disallows AI training and blocks agent crawlers on pages with ads.[^5] A site can be comprehensively unreachable to AI crawlers while every Google-facing report stays clean. The exception is Cloudflare's Block setting for training crawlers, which since 15 September 2026 also applies to mixed-use crawlers, Googlebot included,[^5] so choosing it does turn up as Googlebot errors. [Log file analysis](/technical-seo/log-file-analysis/) is the method; filter by user agent and confirm what each one actually got. ## How should you monitor crawl errors? Treat the Pages report as a trend line, not a daily alarm. Check periodically for sudden jumps in a problem category, a spike in 5xx errors, a climb in soft 404s, a surge in "crawled, not indexed", because a change in the shape of the report is more informative than its absolute numbers. After fixing an issue, use the report's validation flow to ask Google to recheck. For large sites, [log file analysis](/technical-seo/log-file-analysis/) shows what Googlebot actually requested and what it received, giving a fuller picture of crawl failures than Search Console's sampled view. [^1]: [Focusing on the new Search Console — Google Search Central](https://developers.google.com/search/blog/2019/01/focusing-on-new-search-console) [^2]: [Page Indexing report — Google Search Console Help](https://support.google.com/webmasters/answer/7440203) [^3]: [How HTTP status codes affect Google's crawlers — Google Crawling Infrastructure](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes) [^4]: [Reduce the Google crawl rate — Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) [^5]: [Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) [^6]: [Debug network and DNS errors for Google's crawlers — Google Crawling Infrastructure](https://developers.google.com/crawling/docs/troubleshooting/dns-network-errors) [^7]: [Creating helpful, reliable, people-first content — Google Search Central](https://developers.google.com/search/docs/fundamentals/creating-helpful-content) [^8]: [Optimize your crawl budget — Google Crawling documentation](https://developers.google.com/crawling/docs/crawl-budget)