robots.txt and crawlability

Crawlability is the degree to which search engine bots can access the pages of your site. It is the foundation of indexing, and a site with broken crawlability cannot rank no matter how good its content is. robots.txt is the file that controls what bots can and can’t fetch.

How do crawlers discover content?

Search engine crawlers (Googlebot, Bingbot, etc.) discover content through:

• Internal links from already-known pages. The most common discovery path. Pages without internal links are functionally invisible.
• External links from other sites. Backlinks introduce new URLs to crawlers.
• XML sitemaps. Explicit lists of URLs you want indexed.
• Direct submission. Manual URL submission via Search Console (limited use; for individual urgent pages).

Crawlers fetch pages, parse HTML, follow links, and add new URLs to their queue. The process repeats indefinitely, with frequency determined by site authority, change rate, and crawl budget.

What does robots.txt do?

robots.txt is a plain text file at the root of the domain (https://example.com/robots.txt) that tells well-behaved crawlers which URLs they can and cannot fetch. It uses simple syntax:

User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml

User-agent specifies which crawler the rules apply to (* is all). Disallow blocks paths. Allow permits paths within disallowed directories. Sitemap declares one or more sitemap locations.

What is a crawler directive?

A crawler directive is any instruction that tells a search engine bot how it may crawl or index a site. The Disallow, Allow, and User-agent lines above are the crawl-access directives delivered through robots.txt, but the term is broader than one file. It covers three delivery mechanisms, each operating at a different point:

• robots.txt directives control whether a crawler is allowed to fetch a URL at all. They govern access, not indexing.
• Meta robots directives (<meta name="robots" content="noindex">) sit in the page’s HTML and control indexing behaviour, so the page must be crawlable for the directive to be read.
• X-Robots-Tag directives carry the same indexing instructions as the meta robots tag, but in the HTTP response header, which lets them apply to non-HTML files such as PDFs and images.

The common mistake is treating a robots.txt Disallow as an indexing directive. It is not: it stops the fetch, which prevents a crawler from ever seeing a noindex directive on the page. Match the directive to the job, access rules in robots.txt, indexing rules in the meta robots tag or X-Robots-Tag header.

How does a crawler choose which rules to follow?

This is the most misunderstood part of robots.txt, and getting it wrong silently exposes URLs you believe are blocked.

A crawler reads the file, finds the group whose User-agent line most specifically matches its own name, and obeys only that group. Every other group, including the wildcard User-agent: * group, is ignored. Google’s spec is explicit: “User agent specific groups and global groups (*) are not combined.”¹

The * group is a fallback for crawlers that aren’t named anywhere else, not a base layer that named groups inherit from. The moment you give a bot its own group, it stops obeying all the shared * rules.

Consider this file:

User-agent: *
Disallow: /private/
Disallow: /cart/

User-agent: GPTBot
Disallow: /no-ai/

The intuitive reading is “block /private/ and /cart/ for everyone, and also keep GPTBot out of /no-ai/.” That is not what happens. GPTBot matches its own named group, follows only Disallow: /no-ai/, and never consults the * group, so it is free to crawl /private/ and /cart/. The configuration fails open: it exposes the very URLs you meant to protect, with no error to warn you.

To keep shared rules in force for a named crawler, you must repeat them inside that crawler’s group:

User-agent: *
Disallow: /private/
Disallow: /cart/

User-agent: GPTBot
Disallow: /private/
Disallow: /cart/
Disallow: /no-ai/

This is repetitive and scales badly across many named bots, so generate the file from a template rather than editing each group by hand.

Keep two separate ideas distinct:

• Group selection decides which group a crawler obeys. The most specific User-agent match wins, and named groups are never combined with *.
• Rule precedence within that group decides which line wins when several match a URL. The longest matching path wins; on a tie, the least restrictive rule (Allow) wins.

For the full directive and precedence reference, see the robots.txt Reference.

What does robots.txt NOT do?

It does not prevent indexing. Pages disallowed in robots.txt can still appear in search results if Google discovers them through external links. The page won’t be crawled, but the URL might be indexed (with a warning that no description is available). To prevent indexing, use a noindex meta tag, which requires the page to be crawlable.

It does not enforce compliance. robots.txt is a request, not a rule. Well-behaved crawlers (Google, Bing, established AI crawlers) honour it. Malicious or aggressive scrapers ignore it, and some AI agents now decline to honour it as a matter of stated policy. Enforcement, where it exists, happens at the network layer rather than in the file (see crawl control is moving to the CDN below). You can measure how often it is ignored: Microsoft Clarity’s Bot Analytics added a robots.txt violations view in June 2026 that reports non-compliant requests as a share of bot traffic, filterable by bot operator and the URLs targeted.² It requires connecting a supported CDN (Fastly, CloudFront, or Cloudflare).

It does not secure content. Disallowing a path in robots.txt makes its existence public. Anyone can read the file. For genuinely sensitive content, use authentication, not robots.txt.

Common crawlability problems

Accidentally blocking the entire site. A misplaced Disallow: / blocks all crawling. This is the most common (and most damaging) robots.txt mistake. Always verify the live robots.txt after deployment.

Blocking CSS and JavaScript. Googlebot renders pages with their associated CSS and JavaScript to evaluate layout, mobile usability, and content. Blocking these resources damages how Google understands the page. Allow them.

Disallow vs noindex confusion. Disallowing a page in robots.txt while the page also has a noindex meta tag means Google can’t crawl the page to see the noindex directive. The page may end up indexed without a description, the worst of both worlds.

Blocking pagination, faceted filters, or parameterised URLs incorrectly. Some parameter URLs should be canonicalised, some should be noindexed, and some should be allowed. Blanket blocking via robots.txt is rarely the right answer.

Orphan pages. Pages with no internal links are crawled rarely if at all. Sitemaps help, but internal links remain the strongest discovery signal.

Blocking AI training at the CDN and taking Googlebot down with it. A CDN rule that blocks the Training category can also block Googlebot, Bingbot and Applebot, because those crawlers fetch for both search and training and the strictest rule wins. The symptom is a sudden crawl collapse with a robots.txt that looks perfectly correct, because the block never reaches robots.txt. See crawl control is moving to the CDN.

Crawl budget

Crawl budget is the number of URLs Googlebot will crawl on your site within a given time period. For most sites under 10,000 pages, crawl budget is not a constraint. For very large sites (e-commerce with millions of SKUs, news publishers with deep archives), it becomes a real consideration.

The factors that influence crawl budget:

• Site authority. Higher-authority sites get crawled more.
• Site speed. Faster sites get crawled more frequently per second.
• Content freshness. Sites that update frequently get crawled more.
• Discoverable URL count. Larger sites need to share crawl budget across more URLs.

Wasted crawl budget (crawlers fetching low-value URLs at the expense of important ones) is the failure mode. Reduce it by:

• Using robots.txt to block genuinely worthless URLs (admin, search results pages, infinite faceted filter combinations)
• Returning proper status codes for non-existent pages. A soft 404 is a page that returns a 200 OK status but contains no real content: a “not found” message served as a live page, an empty search result, or a removed product page with nothing left on it. Google identifies these algorithmically and may deindex them; they consume crawl budget that should go to real pages. Return 404 or 410 for removed URLs, or restore content so the page earns its 200.
• Consolidating duplicate or near-duplicate content via canonicals
• Cleaning up sitemaps to include only canonical, indexable URLs

AI crawlers

AI crawlers divide into three purposes, and robots.txt can address each separately. The three-way split matters more than it used to, because it is the categorisation Cloudflare now uses to gate access at the network layer (see crawl control is moving to the CDN below).

Training crawlers fetch content to build or update model training datasets. Blocking them prevents your content being used to train LLMs, but has no effect on whether AI search products cite your site.

Search crawlers fetch and index content to power AI search products, in much the same way Googlebot indexes for Search. Blocking them removes your site from those AI search channels.

Agent crawlers fetch a page live, at the moment a user or an agent asks for it, rather than building an index in advance. ChatGPT-User, Claude-User and Perplexity-User sit here. They behave less like a crawler and more like a browser acting on someone’s behalf, which is precisely the argument their operators use for exempting them from robots.txt.

The three decisions are independent. You can block training crawlers while allowing search and agent crawlers, allow all of them, or block all of them.

User agent	Provider	Purpose
Google-Extended	Google	Training
GPTBot	OpenAI	Training
OAI-SearchBot	OpenAI	Search indexing (ChatGPT Search)
ChatGPT-User	OpenAI	Live retrieval
ClaudeBot	Anthropic	Training
Claude-SearchBot	Anthropic	Search indexing (Claude)
Claude-User	Anthropic	Live retrieval
PerplexityBot	Perplexity	Search indexing
Perplexity-User	Perplexity	Live retrieval
Applebot	Apple	Search indexing (Siri, Spotlight) and AI
Applebot-Extended	Apple	Training opt-out only, not a separate crawler3
CCBot	Common Crawl	Training dataset (feeds multiple providers)

Applebot-Extended works differently from the others in the table: it is not a crawler that fetches anything. It exists purely as a token you can disallow to opt out of Apple using content for training, while Applebot itself continues to crawl for search.³

Decision framework:

• You want AI citation: allow the search and agent crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). Training crawlers are irrelevant to citation. Note that OpenAI’s documentation now states that its robots.txt tags apply to OAI-SearchBot and GPTBot, with ChatGPT-User no longer covered;⁴ Perplexity-User explicitly ignores robots.txt by design. Blocking either reliably requires IP-level or WAF rules rather than robots.txt alone.
• You want to limit AI training on your content: block GPTBot, ClaudeBot, Google-Extended, CCBot. This does not affect AI search visibility.
• You are a paywalled publisher: blocking all AI crawlers is a reasonable position. Training crawlers use your content without payment; retrieval crawlers may surface excerpts without sending traffic.
• You have no specific concern: the default (no explicit rules for these agents) allows everything. Most sites are in this position.

A robots.txt configuration that blocks training crawlers while keeping AI search open:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Leaving OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User with no explicit rule allows them, though ChatGPT-User and Perplexity-User may not honour those rules regardless.

AI crawlers represent a measurable and growing share of bot traffic. GPTBot requests grew by more than 300% between May 2024 and May 2025.⁵ Like Googlebot, compliance is voluntary: well-behaved providers honour robots.txt; malicious scrapers do not.

robots.txt vs. the Search Console AI blocking toggle

From June 2026, Google is testing a toggle in Search Console that lets site owners opt their site out of appearing in AI Overviews, AI Mode, and AI Overviews in Discover. This is a different mechanism from robots.txt: blocking Google-Extended stops the training crawler but does not remove your site from AI-generated answers. The Search Console toggle controls appearance in AI search features directly, without affecting crawling or traditional rankings. It launched with UK website owners and is rolling out incrementally to other regions from late June 2026, but is not yet generally available. Opting out does not stop your content powering Search overall or being used for training, which still requires Google-Extended.⁶ See AI Overviews for detail.

robots.txt as an agent-capability pointer

A separate, emerging use of robots.txt is advertising agent-callable capabilities rather than controlling access. The draft Agentic Resource Discovery (ARD) spec, announced in June 2026 by Google, Microsoft, and Hugging Face, defines an Agentmap directive that points to a site’s capability catalog, in the same way a Sitemap directive points to an XML sitemap:

Agentmap: https://example.com/.well-known/ai-catalog.json

This is relevant only to sites that expose MCP servers, agents, or callable APIs, and the spec is an early draft with no confirmed adoption yet. Content and editorial sites have nothing to declare here. See agent-readiness for the wider picture.

Crawl control is moving to the CDN

robots.txt is a request. A CDN block is not. That distinction has stopped being academic.

On 1 July 2026 Cloudflare gave every customer, including free accounts, the ability to sort AI crawlers into the three categories above and manage each separately. From 15 September 2026, Training and Agent crawlers are blocked by default on ad-supported pages for new sites and new customers, while Search crawlers stay allowed.⁷ For a large part of the web, the default answer to “may this AI crawler fetch the page?” is about to change from yes to no, and it will be answered before the request reaches your robots.txt at all.

Two things follow from that.

Your robots.txt may not be the operative control any more. If your site sits behind a CDN that enforces crawler policy, the CDN decides and the file at your document root is a statement of intent. Check what your CDN is configured to do before assuming robots.txt is the whole story.

Blocking training crawlers can block Googlebot. Cloudflare applies the strictest matching rule to crawlers that serve more than one purpose. Googlebot, Bingbot and Applebot all crawl for both search indexing and AI training, so a site that blocks the Training category can find it has blocked the search crawler its traffic depends on.⁷⁸ This is a network-level block rather than a robots.txt line, which makes it harder to spot and harder for the crawler to work around, and it fails in the most expensive direction available: deindexing.

What is the Content-Signal directive?

Cloudflare’s Content Signals policy adds a directive to robots.txt that declares how content may be used, rather than which paths may be fetched:

Content-Signal: search=yes, ai-train=no

Its standing is unsettled, and the honest summary is that nothing acts on it. Google’s John Mueller has said so directly: “It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm.”⁹ No crawler has been shown to honour it and no controlled test has demonstrated an effect. Cloudflare adds it to robots.txt on sites it serves, so you may find it in your own file without having put it there.

The cost of carrying it is a few lines; the demonstrated benefit is currently nil. Treat it as a declaration of preference with no enforcement behind it, in the same category as llms.txt, and do not mistake its presence in your file for protection.

Can crawler identity be enforced?

Not through robots.txt, which has no way to distinguish the real Googlebot from anything that claims to be it. The work aimed at that problem is cryptographic rather than declarative. Web Bot Auth has bots sign their requests with a verifiable key instead of asserting an identity in a header,¹⁰ and the Private Access Control Tokens proposal, developed by Cloudflare with Chrome, Edge and Firefox, applies the same principle to separating legitimate traffic from abuse.

Neither is something you implement in robots.txt today, and neither changes what you should write in that file. They matter because they show where the enforcement problem is being solved: at the network layer, with signatures, rather than in a text file that asks politely.

Auditing crawlability

The basic audit:

1. Read your live robots.txt. Check it is what you expect. Verify nothing critical is disallowed.
2. Use the Google Search Console URL Inspection tool. Spot-check key URLs for crawl status, indexing status, and any blocking signals.
3. Crawl your own site. Tools like Screaming Frog or Sitebulb crawl as Googlebot would and report blocked URLs, redirect chains, broken links, and orphan pages.
4. Review the Crawl Stats report in Search Console. Track crawl rate, response times, and host status over time. Sudden changes warrant investigation.

For a full crawlability checklist, see the Technical SEO Audit Checklist.

Frequently asked questions

Is robots.txt case-sensitive?
Yes. Disallow: /Admin/ and Disallow: /admin/ block different URLs.

Does robots.txt affect rankings?
Indirectly. By controlling what gets crawled and how crawl budget is spent, it influences indexing efficiency and the freshness of indexed content. The file itself is not a ranking signal.

Should I disallow the search results pages on my site?
Generally yes. Internal search results pages are typically low-value, near-infinite in URL combinations, and should not appear in Google’s index. The reliable sequence is to noindex them on-page first, keeping them crawlable so Googlebot can read the directive and drop them from the index. Once they have been deindexed, you can add a robots.txt disallow to save crawl budget. Do not apply both at once: a disallow on its own cannot deindex a page, because Googlebot never reaches the noindex.

Footnotes

1. robots.txt specification — Google Search Central ↩

2. Spot robots.txt violations in Bot Analytics — Microsoft Clarity Blog ↩

3. About Applebot — Apple Support ↩ ↩²

4. OpenAI crawlers — OpenAI developer documentation ↩

5. From Googlebot to GPTBot: who’s crawling your site in 2025 — Cloudflare ↩

6. Manage how your site appears in Search generative AI features — Google Search Help ↩

7. Content Independence Day: bringing AI options to every site — Cloudflare ↩ ↩²

8. Cloudflare’s AI crawler rules can block Googlebot — Search Engine Journal ↩

9. Google on Cloudflare’s Content Signals — Search Engine Roundtable ↩

10. Forget IPs: using cryptography to verify bot and agent traffic — Cloudflare ↩