Guide

When to Use Noindex

Last updated

The noindex directive tells search engines not to include a page in their results. It comes in two forms: a meta tag in the HTML <head>, or an HTTP response header.

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

The HTTP header works for any file type, including PDFs and images, which have no HTML <head>. Both forms do the same thing: once a search engine crawls the page and reads the directive, it removes the page from its index, or never adds it.

The page remains publicly accessible. The URL is not deleted. Anyone who knows the address can still visit it. Noindex only affects whether the page appears in search results.

When noindex is the right call

Thank-you and confirmation pages

A post-purchase or post-submission thank-you page has no independent search value. No one searches for “thank you for your order.” Indexing it creates a page with thin content, no traffic potential, and no purpose beyond a single transactional moment.

Noindex is correct here. The page serves visitors who have already converted; it should not be discoverable through search.

Gated and members-only content

Content behind a login or paywall cannot be served to search engines in full. If you noindex the gated version and maintain an indexable landing page or preview, you control exactly what appears in results. Leaving a gated page crawlable and indexable either presents search engines with a login wall or risks surfacing content you intended to restrict.

Internal search results

A site’s own search results pages (e.g., /search?q=trainers) are typically thin, parameterised, and near-duplicate. Indexing them produces a large volume of low-quality URLs that compete with your actual category and product pages. Noindex keeps crawlers focused on content with real ranking potential.

Faceted navigation and filter URLs

Sites with product filters generate large numbers of URL variants by size, colour, price range, and sort order. Most of these pages are thin, carry no distinct search demand, and duplicate the base category page. Noindex is appropriate for filter combinations that serve no search intent. Some combinations, a specific size and colour of a popular product, for example, may have genuine demand and should be left indexable. Canonical tags are an alternative for consolidating signals rather than removing pages outright.

Staging and development environments

A staging site should never be indexed. It typically contains duplicate content relative to production, may include placeholder copy, and can dilute signals for the live site. Apply a site-wide noindex at CMS level, or combine it with password protection. Robots.txt disallow alone is not sufficient: it blocks crawling but not indexing of any URL Google discovers via links.

Utility pages

Login pages, account dashboards, checkout flows, and cart pages have no search value. They exist for logged-in or transactional users. Noindex keeps them out of results without affecting their function.

The three interactions that cause real damage

Noindex combined with robots.txt disallow

This is the most common and most damaging mistake. It looks like belt-and-braces caution but does the opposite of what is intended.

Noindex only works if Google can crawl the page and read the directive. Robots.txt disallow blocks crawling entirely. Google states the consequence directly: “If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.”1 When both are applied to the same URL, Google respects the disallow, never visits the page, and therefore never reads the noindex. If the URL has inbound links, Google can still index it as a content-free stub: a URL entry with no title and no snippet.

The rule is simple: to keep a page out of results, allow crawling and use noindex. If you want to reduce crawl load for resource reasons, use robots.txt and accept that the URL may still appear in results as a bare stub.

Noindexed URLs in your sitemap

An XML sitemap is a list of URLs you are asking Google to discover and index. A noindexed URL in that list sends two instructions simultaneously: “please index this” (sitemap) and “do not index this” (noindex directive).

Google resolves the conflict by honouring the noindex. But it still crawls the page repeatedly to check the directive, because the sitemap keeps flagging it as a URL to process. In Google Search Console, these pages appear under “Submitted URL marked noindex” in the Page Indexing report. They are not hurting rankings directly, but they consume crawl budget on URLs that can never be indexed. On large sites with misconfigured sitemaps, this measurably reduces crawl frequency for pages that actually matter.

Fix: remove noindexed URLs from your sitemap. Only include canonical, indexable URLs returning a 200 response.

Noindex on a canonical target

If page A has a canonical tag pointing to page B, and page B carries a noindex directive, neither page will be indexed. The noindex on the canonical target wins. Googlebot reads page A’s canonical, follows it to page B, reads the noindex, and excludes both.

This pattern appears most often after a site reorganises its indexing strategy and noindexes a URL that other pages still point to canonically. Audit with a crawl tool to surface canonical chains that terminate at noindexed pages.

Noindex and nofollow are independent

A common assumption: noindexing a page stops PageRank flowing through links on that page. This is not correct.

Noindex tells Google not to include the page in results. Nofollow tells Google not to follow links on that page. They are separate directives and each must be declared explicitly. Google also accepts none as shorthand, which it documents as “equivalent to noindex, nofollow”.1

<meta name="robots" content="noindex, nofollow">

For most noindex use cases (thank-you pages, utility pages), adding nofollow is sensible. For pages you noindex for content quality reasons while still wanting internal links followed, such as some faceted navigation scenarios, use noindex alone.

With one caveat that changes the plan on a long enough timescale. A page left on noindex, follow indefinitely does not keep passing link signals forever. John Mueller has explained that long-term noindex, follow ends up equivalent to noindex, nofollow, because Google eventually drops the page from the index and crawls it progressively less, so the links on it stop being followed in practice.2 noindex, follow is therefore a reasonable transitional state, and a poor permanent architecture. If a page’s only job is to pass internal link equity to others, the durable answers are a crawlable indexable hub, a canonical, or restructured internal linking, not a permanently noindexed page.

Noindex and AI crawlers

The name="robots" meta tag is a generic instruction that any crawler may choose to honour. Narrowing it to a specific crawler is where the mechanism gets misunderstood, and the misunderstanding is worth stating plainly because it circulates as copy-and-paste code.

Per-crawler meta tags are not a general mechanism. Google supports exactly two user agent tokens in the robots meta tag, googlebot and googlebot-news, and says of the rest: “Google supports two user agent tokens in the robots meta tag; other values are ignored.”1 A token is only meaningful if the vendor operating that crawler has documented support for it.

<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">

The major AI crawlers document robots.txt, not meta tags. OpenAI’s crawler documentation describes robots.txt entries for GPTBot, OAI-SearchBot and ChatGPT-User, and documents no meta tag.3 Anthropic’s documentation for ClaudeBot is the same: a User-agent: ClaudeBot / Disallow: / block in robots.txt, plus support for the non-standard Crawl-delay extension, with no meta tag offered.4

Tags of the form <meta name="GPTBot" content="noindex"> are not a documented control and should not be relied on. Use robots.txt for these crawlers.

The wider picture is also less settled than for search engines. Behaviour differs between crawlers used for live retrieval and those used for training-data collection, and compliance is not uniform. Blocking a training crawler does not remove content already in a model, and does not necessarily stop the page being fetched at answer time by a different agent.

From June 2026, Google is also testing a Search Console toggle that lets site owners opt out of appearing in AI Overviews, AI Mode, and AI Overviews in Discover.5 It launched to a small subset of UK site owners, because the CMA required Google to give publishers an opt-out, and has been extending to other regions since; it is not generally available. This is separate from noindex: the toggle controls appearance in AI search features without affecting indexing or traditional rankings. See AI Overviews for detail.

How to audit for accidental noindex

Accidental noindex is one of the most common findings in a technical SEO audit. It causes complete loss of visibility, which is worse than most ranking problems, and it can persist unnoticed for months.

Google Search Console Page Indexing report. Go to Indexing > Pages and look at the “Excluded by noindex tag” group. Any URL listed there that should be ranking is a priority fix.

CMS settings. WordPress has a site-wide “Discourage search engines” toggle in Settings > Reading. It applies noindex across the entire site and occasionally gets enabled accidentally during development or staging. Shopify, Squarespace, and other platforms have equivalent settings. Check these first on any site that has recently had development work or gone through a platform migration.

Site crawl. Screaming Frog, Sitebulb, and similar tools export a full list of noindexed pages found during a crawl. Cross-reference this against pages that should be indexable.

View source. On any individual page, search the page source for “noindex.” Check both the <head> section (meta tag) and the HTTP response headers (x-robots-tag).

Post-migration check. Staging environments almost always carry a site-wide noindex. The single most common migration mistake is launching without removing it. Rankings can drop before the error surfaces clearly in GSC.

Removing noindex: the recovery mistake

A page incorrectly set to noindex for weeks or months may not re-index immediately after you remove the directive. If Googlebot has crawled the page repeatedly and found noindex each time, it may eventually deprioritise that URL to conserve crawl budget. As a result it may be crawled infrequently, meaning the removal could take days or months to register. Simply removing the meta tag or header won’t trigger a re-crawl fast enough.

Manually submit to GSC

After removing noindex from a page, manually submit it to Google Search Console to re-queue it for crawling. This signals that the page is now worth processing, rather than waiting for Googlebot's normal crawl cycle to discover the change. Without resubmission, recovery can take weeks.

Decision guide

Page typeRecommendation
Thank-you / confirmationNoindex
Gated or members-onlyNoindex
Internal search resultsNoindex
Login / account / checkoutNoindex
Staging environmentNoindex + password protection
Filter / faceted navigationNoindex if no distinct search demand; leave indexable if specific combination has search volume
Paginated pages beyond page 1Noindex if thin; leave indexable if pages contain content with real search demand
Duplicate parameter URLsCanonical to the clean URL; noindex if no canonical target makes sense
Thin or low-quality contentFix the content or consolidate with a redirect. Noindex is not a substitute
Any page you want to rankDo not noindex

Footnotes

  1. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — Google Search Central ↩ ↩2 ↩3

  2. Google: Long Term Noindex Will Lead To Nofollow On Links — Search Engine Roundtable, 28 December 2017. Mueller’s remarks come from a webmaster hangout at the 54:51 mark. ↩

  3. Bots — OpenAI. Documents robots.txt control for GPTBot, OAI-SearchBot and ChatGPT-User. ↩

  4. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic ↩

  5. Google Search Console AI Performance Report & AI Blocking Controls (Limited) — Search Engine Roundtable, 3 June 2026. Records the UK-first rollout and the CMA requirement behind it. ↩