Indexing
Last updated
Indexing is the last stage of the crawl, render, and index pipeline: after Google fetches the raw HTML and processes the JavaScript, it decides whether to add the page to its searchable database. Crawling and indexing are separate decisions, and the gap between them is one of the most misunderstood areas of technical SEO.
A page can be crawled consistently and never indexed. The crawl is access. Indexing is an editorial decision, and Google says so plainly: “Indexing isn’t guaranteed; not every page that Google processes will be indexed”, with low content quality the first reason it lists.1 Pages that violate Google’s content policies may be restricted from standard results regardless of other quality signals. SafeSearch is a different mechanism and does not belong in that sentence: it filters at display time for users who have it on, and the page stays fully indexed and ranks normally for everyone else.
Index membership is also the precondition for Google’s AI surfaces: AI Overviews and AI Mode draw their answers from pages in Google’s index, so a page Google has not indexed cannot be surfaced or cited there.
See How Google crawls, renders, and indexes pages for a full description of the pipeline that precedes this decision.
What does Google evaluate at the indexing stage?
After crawling and rendering a page, Google assesses whether it belongs in the index. Several factors influence that decision.
Content uniqueness and standalone value
Google’s index covers “hundreds of billions of web pages” and is “well over 100,000,000 gigabytes” in size.2 A new page needs to offer something the index does not already have, or to offer it better. Pages that closely mirror existing indexed content, add no distinct perspective, or answer a query no differently from a dozen already-indexed pages are candidates for exclusion.
This is not about originality in the creative sense. A factual page covering a well-worn topic can still earn indexing if it is accurate, well-structured, and serves the query better than what exists. The question Google is effectively asking: does this page help a user in a way that justifies its place in the index?
Thin content
Thin content is content with insufficient unique substance relative to what already exists in the index. It is not purely a word-count issue. A 2,000-word page built from padding is thin. A 400-word page that directly answers a specific question may not be.
Common sources of thin content: template-generated pages with minimal variation, category or tag pages with no editorial content, product pages with only manufacturer descriptions, and location pages using the same copy for every city.
noindex directive
A <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header explicitly instructs Google not to index the page. The page must be crawlable for Google to read this directive. A page blocked in robots.txt cannot be noindexed this way. Google’s own caution is explicit: if the page is blocked, “the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it”.3
Robots.txt controls crawling only, not indexing
Two things to know about robots.txt and indexing: Google stopped obeying noindex directives placed inside robots.txt in 2019 (Google Search Central). That technique no longer works. And if a page is disallowed in robots.txt, Googlebot cannot reach it, so any noindex tag in the page's HTML is never seen, meaning the page can still be indexed if external links point to it. Robots.txt is only a crawl gate. To prevent indexing, use a meta robots noindex tag or an X-Robots-Tag HTTP header on a page that Googlebot can actually access.
Canonical signals
If a page declares a canonical tag pointing to a different URL, Google consolidates signals on that URL and typically does not index the source page independently. See canonical tags for how this works and the failure patterns to avoid.
HTTP status codes
A 200 passes the content to the indexing pipeline, though Google is clear that “the indexing systems may index the content, but that’s not guaranteed”. A 404 or 410 signals that the content does not exist, and Google “removes the URL from the index if it was previously indexed”. A 503 is not an indefinite reprieve: Google decreases the crawl rate, and its “indexing pipeline removes from the index URLs that persistently return a server error”. A page returning 200 with empty or error content is reported as a soft 404 and may also be excluded.4
Authority as a soft indexing input
Google’s indexing is not uniform across all sites, and the documented part of the reason is crawl demand rather than indexing itself. Google says that for Googlebot, demand “varies based on a site’s size, update frequency, page quality, and relevance, compared to other sites”, and that “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems”.5 A page crawled sooner and revisited more often simply gets assessed more often, so a new page on an established site can appear in the index within hours while the same page elsewhere waits days or weeks.
What follows from that is practitioner inference, and worth marking as such. The widely-held view is that a site carrying a large proportion of low-quality pages faces a higher bar across the whole domain, not just on the weak pages. Google has not stated it. The remedy it points to, improving the overall quality of what the site publishes, is sound either way, which is the same reasoning deindexing applies to the parallel claim that Google has grown more selective about what it retains.
Google’s Page Indexing report
The Page Indexing report in Google Search Console (Indexing > Pages) shows every URL Google has discovered on a site, split between indexed and not indexed. Non-indexed URLs are grouped by reason code. The counts are authoritative, but the URL lists are not: Google limits the examples to 1,000 items and says the list “isn’t guaranteed to show all URLs in a given status, even when less than 1,000 items”.6 Treat the visible URLs as a sample at any size, not only on large sites.
The most common reason codes and what they indicate:
| Reason code | What it means |
|---|---|
| Crawled - currently not indexed | Google fetched the page but chose not to index it. Content quality is the usual cause. |
| Discovered - currently not indexed | Google found the URL but has not yet crawled it. Often a prioritisation or crawl budget issue. |
| Duplicate without user-selected canonical | Google found duplicates and chose a canonical itself. Add explicit canonicals. |
| Duplicate, Google chose different canonical than user | Your declared canonical was overridden. Investigate conflicting signals. |
| URL marked ‘noindex’ | The page has a noindex directive. Intentional or accidental. |
| Alternate page with proper canonical tag | The page canonicals to a different URL. Indexing consolidated there. |
| Blocked by robots.txt | Googlebot could not crawl the page to assess it. |
| Not found (404) | The page does not exist or returns a 404. |
For a full diagnostic walkthrough of each reason code and how to fix it, see the why isn’t my page indexed guide.
How do you accelerate indexing?
Google does not provide a general-purpose mechanism for forcing indexing. The most reliable signals are indirect.
Internal linking. Pages with strong internal links from already-indexed pages are discovered and prioritised for crawling faster. A new page with no internal links may sit undiscovered for weeks. Add links from topically relevant, well-trafficked pages.
XML sitemap. Include new pages in the sitemap and keep it current. The sitemap signals to Google that you consider these URLs important. Submit it via Search Console under Indexing > Sitemaps.
External backlinks. An external link from a high-authority domain is one of the fastest ways to get a new page indexed. When Googlebot crawls the linking page and finds the link, it adds the linked URL to its crawl queue.
URL Inspection request. In Search Console, the URL Inspection tool has a “Request indexing” button. This queues the URL for crawling and is useful for individual high-priority pages after a significant change. It does not guarantee indexing; it is not a substitute for the signals above.
Request indexing sparingly, and don't count on a fixed reset
The Request indexing button has a per-property daily quota, widely observed at around 10 URLs per day, though Google doesn't publish the figure. It runs on a rolling 24-hour window rather than resetting at a fixed time, so if you hit the cap you may free up capacity an hour or two later. Reserve it for individual high-priority pages after a real change. It is not the Indexing API described below, which has a separate quota and only accepts JobPosting and BroadcastEvent content.
Indexing API
Google’s Indexing API is a programmatic way to notify Google of URL updates. Google restricts it to two content types: it “can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject”.7 General-purpose web content cannot be submitted through this API. Attempting to use it for unsupported content types has no effect.
IndexNow
IndexNow is a protocol developed by Microsoft that lets sites notify search engines of URL additions, updates, and deletions in real time, without waiting for the next crawl. It works for any content type, and the protocol’s own registry currently lists seven endpoints: Bing, Yandex, Naver, Seznam.cz, Yep, the Internet Archive and Amazon’s crawler.8 The last two are not search engines, which is covered on the IndexNow page.
Google does not currently participate in IndexNow. For sites optimising for Bing alongside Google, IndexNow can accelerate Bing discovery significantly. For Google specifically, the indirect signals above remain the most reliable approach.
Frequently asked questions
Why is my new page not getting indexed?
The most common reasons are insufficient internal links (Google has not prioritised crawling it) and content quality concerns (Google crawled it but decided it did not belong in the index). Check the Page Indexing report in Search Console to see the specific reason code for the URL.
How long does indexing take?
It varies substantially. High-authority sites with strong internal linking can see new pages indexed within hours. Lower-authority sites with weak internal linking may wait days to weeks. Pages that are crawled but not indexed due to quality signals may never be indexed without editorial changes to the content.
Can I force Google to index a page?
No. The URL Inspection request in Search Console queues a page for crawling; it does not guarantee indexing. The underlying signals, content quality, internal links, and authority, determine whether the page is indexed after it is crawled.
Footnotes
-
In-depth guide to how Google Search works — Google Search Central ↩
-
Block Search indexing with noindex — Google Search Central. The same page states that specifying
noindexin robots.txt “is not supported by Google”. ↩ -
How HTTP status codes, and network and DNS errors affect Google Search — Google Search Central ↩
-
Crawl budget management for large sites — Google Search Central ↩
-
IndexNow participating search engines registry. Seven endpoints as at 10 September 2026. ↩
