Crawl Errors

A crawl error is what happens when Google tries to fetch one of your URLs and cannot. The fetch may fail because your server returned an error, the domain would not resolve, a redirect looped, or a directive blocked access. Crawl errors matter because a page Google cannot crawl cannot be indexed, and a pattern of them can slow crawling across the whole site. The skill is less in finding them than in telling a real failure from a normal, intentional exclusion.

This is the diagnostic layer beneath indexing: indexing is whether a page makes it into Google’s index, and crawl errors are one of the reasons a page never gets the chance.

Where do you find crawl errors now?

Google retired the standalone Crawl Errors report in 2019.¹ Crawl and indexing problems now live in the Pages report (formerly Index Coverage) inside Google Search Console. It groups URLs into indexed and “not indexed”, and lists the reason for each non-indexed group, along with warnings for pages that are indexed despite an issue.

One thing the Pages report does not surface as a reason code is a DNS or connectivity failure. Those show up through URL Inspection and the Crawl Stats report instead, so a site that cannot be reached at all will not necessarily announce itself in the place you are looking.

The URL Inspection tool complements the report: for any single URL it shows the last crawl outcome, whether the page is indexed, and what Google encountered, which is how you diagnose a specific page rather than a category.

What are the main types of crawl error?

Server errors (5xx). Your server returned a 500, 502, 503, or similar when Google requested the page. These are genuine problems: they block crawling and, at scale, signal an unreliable site that Google will crawl more cautiously. Persistent 5xx errors are a priority fix.

DNS and connectivity failures. Google could not resolve your domain or reach the server at all. Usually intermittent hosting or DNS issues; if sustained, they are serious because they can affect the whole site.

Redirect errors. Redirect chains that are too long, loops, or redirects to broken URLs. These waste crawl budget and can strand pages. Resolve with clean, single-hop redirects.

404 (not found). A requested URL does not exist. A 404 is not inherently a problem, removed pages should 404 or 410, but a 404 that receives internal links or external links is a leak worth fixing.

Soft 404. The page returns a 200 status but has no meaningful content, or tells the user the content is missing. Google treats it as a not-found in disguise. These need a real fix: either restore content, return a proper 404, or redirect to a relevant page.

Blocked URLs. The URL is disallowed in robots.txt. Whether this is an error depends entirely on intent, which is the heart of triage below. A noindex page is a different case and not a crawl error at all: it was fetched successfully, and the exclusion is an indexing decision rather than a fetch failure.

Which status codes actually slow Googlebot down?

This is where intuition misleads people, so it is worth stating Google’s documented behaviour directly. Only two things reduce crawl rate: 5xx server errors, and 429. Google is explicit that “Googlebot treats the 429 status code as a signal that the server is overloaded, and it’s considered a server error”, and that 5xx and 429 together “prompt Google’s crawlers to temporarily slow down with crawling”.²

Everything else in the 4xx family does not. In Google’s words, “the 4xx status codes, except 429, have no effect on crawl rate”, and it warns directly against the common workaround: “Don’t use 401 and 403 status codes for limiting the crawl rate.”² All 4xx except 429 are treated identically, signalling to the indexing pipeline that the content does not exist, which removes the URL if it was previously indexed.

Two practical consequences. A wall of 403s from a bot-mitigation rule will not throttle Googlebot at all; it will deindex the affected URLs instead, which is worse, and it will do so without any signal that reads as a crawl-rate problem. And if you genuinely need to shed crawl load during an incident, 503 or 429 is the correct response, because both preserve indexed URLs while you recover. That is a short-term measure: Google warns that returning unavailability codes for more than a few days will cause it to slow or stop crawling those URLs permanently.³

How do you tell a real error from an intentional exclusion?

Most of the “not indexed” volume in a healthy site’s Pages report is deliberate and correct. Treating every non-indexed URL as a problem is the most common way to waste time here. Work through the reasons and classify each:

The reason names below are the labels Google uses in the report today. They have been renamed more than once, so older guides quote strings you will not find in the interface.⁴

Almost always intentional and fine:

• URL marked ‘noindex’: you asked for this on pages you do not want indexed (thank-you pages, filtered views).
• Alternate page with proper canonical tag: Google chose your declared canonical; the duplicate is correctly folded in.
• Blocked by robots.txt: expected for admin, cart, or parameter URLs you deliberately disallowed.
• Page with redirect: the old URL redirects; it should not be indexed.

Investigate, often a real problem:

• Server error (5xx): fix these.
• Soft 404: fix content, status, or redirect.
• Blocked due to unauthorized request (401), Blocked due to access forbidden (403) and URL blocked due to other 4xx issue: usually a bot-mitigation rule, WAF or rate limiter treating Googlebot as hostile traffic rather than anything you configured deliberately. Google is direct about the 403 case: “Googlebot never provides credentials, so your server is returning this error incorrectly.”⁴
• Discovered - currently not indexed at scale: Google knows the URL but has not crawled it. Google’s stated cause is capacity, not quality: “Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.”⁴ Thin content and weak internal linking are plausible secondary readings, but check crawl budget and server response first.
• Crawled - currently not indexed at scale: Google crawled but judged the page not worth indexing; usually a quality or duplication signal, not a technical one.
• Duplicate without user-selected canonical and Duplicate, Google chose different canonical than user: worth checking your canonical signals are consistent.

The rule of thumb: a handful of a given reason is usually noise; a large or growing count of a problem reason is a signal. Fix the genuine failures, confirm the exclusions are deliberate, and resist driving every non-indexed number to zero, which is neither achievable nor desirable.

How should you monitor crawl errors?

Treat the Pages report as a trend line, not a daily alarm. Check periodically for sudden jumps in a problem category, a spike in 5xx errors, a climb in soft 404s, a surge in “crawled, not indexed”, because a change in the shape of the report is more informative than its absolute numbers. After fixing an issue, use the report’s validation flow to ask Google to recheck. For large sites, log file analysis shows what Googlebot actually requested and what it received, giving a fuller picture of crawl failures than Search Console’s sampled view.

Footnotes

1. Focusing on the new Search Console — Google Search Central ↩

2. How HTTP status codes, and network and DNS errors affect Google Search — Google Search Central ↩ ↩²

3. Large site owner’s guide to managing your crawl budget — Google Search Central ↩

4. Page Indexing report — Google Search Console Help ↩ ↩² ↩³