--- title: "robots.txt and crawlability" description: "How search engine crawlers discover and access your site, what robots.txt does and doesn't do, and common ways crawl is accidentally broken." tldr: "Crawlability means search engine bots can fetch a site's pages, and robots.txt, a plain-text file at the site root, tells them which URLs they may request. It does not prevent indexing: a disallowed URL can still appear in search results if it has inbound links. Common crawl errors are blocks in misconfigured robots.txt, noindex tags on pages that should be indexed, and canonical tags pointing the wrong way. For AI crawlers, control is shifting to the CDN: since 15 September 2026 Cloudflare's Block setting on the Training category also stops Googlebot, so the option that keeps search crawling is Disallow AI Training." publishDate: 2026-04-25 updatedDate: 2026-10-19 author: "Liam Hayward" --- Crawlability is the degree to which search engine bots can access the pages of your site. It is the foundation of indexing, and a site with broken crawlability cannot rank no matter how good its content is. `robots.txt` is the file that controls what bots can and can't fetch. ## How do crawlers discover content? Search engine crawlers (Googlebot, Bingbot, etc.) discover content through: - **Internal links from already-known pages.** Pages without internal links are functionally invisible. - **External links from other sites.** Backlinks introduce new URLs to crawlers. - **XML sitemaps.** Explicit lists of URLs you want indexed. - **Direct submission.** Manual URL submission via Search Console (limited use; for individual urgent pages). Crawlers fetch pages, parse HTML, follow links, and add new URLs to their queue. The process repeats indefinitely. Google documents how often it recrawls a URL as depending on how popular the URL is and how stale its copy is, within the crawl capacity the server can sustain.[^14] ## What does robots.txt do? `robots.txt` is a plain text file at the root of the domain (`https://example.com/robots.txt`) that tells well-behaved crawlers which URLs they can and cannot fetch. It uses simple syntax: ``` User-agent: * Disallow: /admin/ Disallow: /search/ Allow: / User-agent: Googlebot Disallow: /no-google/ Sitemap: https://example.com/sitemap.xml ``` `User-agent` specifies which crawler the rules apply to (`*` is all). `Disallow` blocks paths. `Allow` permits paths within disallowed directories. `Sitemap` declares one or more sitemap locations. ## What is a crawler directive? A crawler directive is any instruction that tells a search engine bot how it may crawl or index a site. The `Disallow`, `Allow`, and `User-agent` lines above are the crawl-access directives delivered through robots.txt, but the term is broader than one file. It covers three delivery mechanisms, each operating at a different point: - **robots.txt directives** control whether a crawler is allowed to *fetch* a URL at all. They govern access, not indexing. - **Meta robots directives** (``) sit in the page's HTML and control *indexing* behaviour, so the page must be crawlable for the directive to be read. - **X-Robots-Tag directives** carry the same indexing instructions as the meta robots tag, but in the HTTP response header, which lets them apply to non-HTML files such as PDFs and images. The common mistake is treating a robots.txt `Disallow` as an indexing directive. It is not: it stops the fetch, which prevents a crawler from ever seeing a `noindex` directive on the page. Match the directive to the job, access rules in robots.txt, indexing rules in the meta robots tag or `X-Robots-Tag` header. ## How does a crawler choose which rules to follow? This is a commonly misunderstood part of robots.txt, and getting it wrong silently exposes URLs you believe are blocked. A crawler reads the file, finds the group whose `User-agent` line most specifically matches its own name, and obeys **only that group**. Every other group, including the wildcard `User-agent: *` group, is ignored. Google's spec is explicit: "User agent specific groups and global groups (`*`) are not combined."[^2] The `*` group is a fallback for crawlers that aren't named anywhere else, not a base layer that named groups inherit from. The moment you give a bot its own group, it stops obeying all the shared `*` rules. Consider this file: ``` User-agent: * Disallow: /private/ Disallow: /cart/ User-agent: GPTBot Disallow: /no-ai/ ``` The intuitive reading is "block `/private/` and `/cart/` for everyone, and also keep GPTBot out of `/no-ai/`." That is not what happens. GPTBot matches its own named group, follows only `Disallow: /no-ai/`, and never consults the `*` group, so it is free to crawl `/private/` and `/cart/`. The configuration fails open: it exposes the very URLs you meant to protect, with no error to warn you. To keep shared rules in force for a named crawler, you must **repeat them inside that crawler's group**: ``` User-agent: * Disallow: /private/ Disallow: /cart/ User-agent: GPTBot Disallow: /private/ Disallow: /cart/ Disallow: /no-ai/ ``` This is repetitive and scales badly across many named bots, so generate the file from a template rather than editing each group by hand. Keep two separate ideas distinct: - **Group selection** decides which group a crawler obeys. The most specific `User-agent` match wins, and named groups are never combined with `*`. - **Rule precedence within that group** decides which line wins when several match a URL. The longest matching path wins; on a tie, the least restrictive rule (`Allow`) wins. For the full directive and precedence reference, see the [robots.txt Reference](/resources/robots-txt-reference/). ## What does robots.txt NOT do? **It does not prevent indexing.** Pages disallowed in robots.txt can still appear in search results if Google discovers them through external links. The page won't be crawled, but the URL might be indexed (with a warning that no description is available). To prevent indexing, use a `noindex` meta tag, which requires the page to be crawlable. **It does not enforce compliance.** robots.txt is a request, not a rule. Well-behaved crawlers (Google, Bing, established AI crawlers) honour it. Malicious or aggressive scrapers ignore it, and some AI agents now decline to honour it as a matter of stated policy. Enforcement, where it exists, happens at the network layer rather than in the file (see [crawl control is moving to the CDN](#crawl-control-is-moving-to-the-cdn) below). You can measure how often it is ignored: Microsoft Clarity's Bot Analytics added a robots.txt violations view in June 2026 that reports non-compliant requests as a share of bot traffic, filterable by bot operator and the URLs targeted.[^3] It requires connecting a supported CDN (Fastly, CloudFront, or Cloudflare). **It does not secure content.** Disallowing a path in robots.txt makes its existence public. Anyone can read the file. For sensitive content, use authentication, not robots.txt. ## Common crawlability problems **Accidentally blocking the entire site.** A misplaced `Disallow: /` blocks all crawling. This is a common robots.txt mistake and, because it blocks everything, the most damaging one. Always verify the live robots.txt after deployment. **Blocking CSS and JavaScript.** Googlebot renders pages with their associated CSS and JavaScript to evaluate layout, mobile usability, and content. Blocking these resources damages how Google understands the page. Allow them. **Disallow vs noindex confusion.** Disallowing a page in robots.txt while the page also has a noindex meta tag means Google can't crawl the page to see the noindex directive. The page may end up indexed without a description, the worst of both worlds. **Blocking pagination, faceted filters, or parameterised URLs incorrectly.** Some parameter URLs should be canonicalised, some should be noindexed, and some should be allowed. Blanket blocking via robots.txt is rarely the right answer. **Orphan pages.** Pages with no internal links are crawled rarely if at all. Sitemaps help, but they do not replace internal links. **Blocking AI training at the CDN and taking Googlebot down with it.** A CDN rule that blocks the Training category can also block Googlebot, Bingbot and Applebot, because those crawlers fetch for both search and training and Cloudflare's Block setting now applies to them; Disallow AI Training is the setting that keeps them crawling for search. The symptom is a sudden crawl collapse with a robots.txt that looks perfectly correct, because the block never reaches robots.txt. See [crawl control is moving to the CDN](#crawl-control-is-moving-to-the-cdn).

Don't block CSS and JavaScript in robots.txt

Blocking your CSS, JavaScript, or theme directories stops Google rendering the page as a user sees it, which can distort how it understands your content and layout. It is easier to do this by accident than people realise. A rule meant to keep parameter URLs out of the index, such as Disallow: /*?, also blocks any script or stylesheet loaded with a cache-busting query string like /theme/main.css?v=8f2a1c, so the page renders unstyled to Googlebot. The same happens with a broad Disallow: /assets/ or Disallow: /wp-content/ that catches the render-critical files inside. Allow crawling of everything needed to render the page, and reserve disallows for low-value paths, not the assets that build the page.

## Crawl budget Crawl budget is the number of URLs Googlebot will crawl on your site within a given time period. For most sites under 10,000 pages, crawl budget is not a constraint: Google writes its crawl budget guidance for sites with over a million pages that change weekly, or over 10,000 that change daily.[^14] For very large sites (e-commerce with millions of SKUs, news publishers with deep archives), it becomes a real consideration. The factors Google documents:[^14] - **Crawl health.** If the site responds consistently and quickly, Google raises the number of connections it uses; slowdowns, 5xx errors and 429 responses lower it. - **Popularity.** URLs that are more popular on the web tend to be crawled more often. - **Staleness.** Google recrawls often enough to pick up changes. - **Perceived inventory.** Without guidance, Google tries to crawl most of the URLs it knows about, so duplicates and unwanted URLs take a share of the crawl. Wasted crawl budget (crawlers fetching low-value URLs at the expense of important ones) is the failure mode. Reduce it by: - Using robots.txt to block worthless URLs (admin, search results pages, infinite faceted filter combinations) - Returning proper status codes for non-existent pages. A soft 404 is a page that returns a 200 OK status but contains no real content: a "not found" message served as a live page, an empty search result, or a removed product page with nothing left on it. Google identifies these algorithmically and may deindex them; they consume crawl budget that should go to real pages. Return 404 or 410 for removed URLs, or restore content so the page earns its 200. - Consolidating duplicate or near-duplicate content via canonicals - Cleaning up sitemaps to include only canonical, indexable URLs ## AI crawlers AI crawlers divide into three purposes, and robots.txt can address each separately. The three-way split matters more than it used to, because it is the categorisation Cloudflare now uses to gate access at the network layer (see [crawl control is moving to the CDN](#crawl-control-is-moving-to-the-cdn) below). **Training crawlers** fetch content to build or update model training datasets. Blocking them prevents your content being used to train LLMs, but has no effect on whether AI search products cite your site: OpenAI documents GPTBot and OAI-SearchBot as separate crawlers with separate robots.txt controls, and says only the second decides whether a site appears in ChatGPT search.[^5] **Search crawlers** fetch and index content to power AI search products, in much the same way Googlebot indexes for Search. Blocking them removes your site from those AI search channels. **Agent crawlers** fetch a page live, at the moment a user or an agent asks for it, rather than building an index in advance. `ChatGPT-User`, `Claude-User` and `Perplexity-User` sit here. They behave less like a crawler and more like a browser acting on someone's behalf, which is precisely the argument their operators use for exempting them from robots.txt. The three decisions are independent. You can block training crawlers while allowing search and agent crawlers, allow all of them, or block all of them. | User agent | Provider | Purpose | |---|---|---| | `Google-Extended` | Google | Training and Gemini Apps grounding opt-out only, not a separate crawler[^15] | | `GPTBot` | OpenAI | Training | | `OAI-SearchBot` | OpenAI | Search indexing (ChatGPT Search) | | `ChatGPT-User` | OpenAI | Live retrieval | | `ClaudeBot` | Anthropic | Training | | `Claude-SearchBot` | Anthropic | Search indexing (Claude) | | `Claude-User` | Anthropic | Live retrieval | | `PerplexityBot` | Perplexity | Search indexing | | `Perplexity-User` | Perplexity | Live retrieval | | `Applebot` | Apple | Search indexing (Siri, Spotlight) and AI | | `Applebot-Extended` | Apple | Training opt-out only, not a separate crawler[^9] | | `CCBot` | Common Crawl | Training dataset (feeds multiple providers) | `Google-Extended` and `Applebot-Extended` work differently from the others in the table: neither is a crawler that fetches anything. Each is a token you can disallow while the company's main crawler keeps crawling for search. `Google-Extended` covers training of Gemini models and grounding in Gemini Apps and Vertex AI, with fetching done by Google's existing crawlers;[^15] `Applebot-Extended` covers training of Apple's foundation models.[^9] **Decision framework:** - **You want AI citation:** allow the search and agent crawlers (`OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`, `Perplexity-User`). Training crawlers are irrelevant to citation. Note that OpenAI's documentation names `OAI-SearchBot` and `GPTBot` as its robots.txt tags and says that for `ChatGPT-User`, because its fetches are user-initiated, "robots.txt rules may not apply";[^5] Perplexity says `Perplexity-User` "generally ignores robots.txt rules" for the same reason.[^16] Blocking either reliably requires IP-level or WAF rules rather than robots.txt alone. - **You want to limit AI training on your content:** block `GPTBot`, `ClaudeBot`, `Google-Extended`, `CCBot`. This does not affect visibility in AI search products or in Google Search, with one exception: `Google-Extended` also withdraws your content from grounding in Gemini Apps.[^15] - **You are a paywalled publisher:** blocking all AI crawlers is a reasonable position. Training crawlers use your content without payment; retrieval crawlers may surface excerpts without sending traffic. - **You have no specific concern:** the default (no explicit rules for these agents) allows everything. Most sites are in this position. A robots.txt configuration that blocks training crawlers while keeping AI search open: ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / ``` Leaving `OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`, and `Perplexity-User` with no explicit rule allows them, though `ChatGPT-User` and `Perplexity-User` may not honour those rules regardless. AI crawlers represent a measurable and growing share of bot traffic. GPTBot requests grew by more than 300% between May 2024 and May 2025.[^1] Like Googlebot, compliance is voluntary: well-behaved providers honour robots.txt; malicious scrapers do not. ## How does the Search Console AI toggle differ from robots.txt? Search Console carries a toggle that lets site owners opt their site out of appearing in AI Overviews, AI Mode, and the generative AI features in Discover. This is a different mechanism from robots.txt: blocking `Google-Extended` limits training and Gemini Apps grounding but does not remove your site from AI Overviews or AI Mode. The Search Console toggle controls appearance in AI search features directly, without affecting crawling or traditional rankings. It launched with UK website owners in June 2026 and reached all websites worldwide on 31 August 2026. Opting out does not stop your content powering Search overall or being used for training, which still requires `Google-Extended`.[^4] See [AI Overviews](/ai-search/ai-overviews/#how-do-you-opt-out-of-ai-overviews) for detail. ## Can robots.txt advertise agent capabilities? A separate, emerging use of robots.txt is advertising agent-callable capabilities rather than controlling access. The proposed Agentic Resource Discovery (ARD) spec, announced in June 2026 by Google, Microsoft, and Hugging Face, defines an `Agentmap` directive that points to a site's capability catalog, in the same way a `Sitemap` directive points to an XML sitemap: ``` Agentmap: https://example.com/.well-known/ard.json ``` This is relevant only to sites that expose MCP servers, agents, or callable APIs, and the spec is an early proposal with no confirmed adoption yet. Its manifest was named `ai-catalog.json` until version 0.91 in August 2026, so older examples of this line point there.[^13] Content and editorial sites have nothing to declare here. See [agent-readiness](/ai-search/agent-readiness/) for the wider picture. ## Crawl control is moving to the CDN robots.txt is a request. A CDN block is not. That distinction has stopped being academic. On 1 July 2026 Cloudflare gave every customer, including free accounts, the ability to sort AI crawlers into the three categories above and manage each separately. It announced that from **15 September 2026** Training and Agent crawlers would be blocked by default on ad-supported pages.[^6] What shipped on that date is a recommended preset for newly onboarded domains: a site that says it earns from ads gets Training set to Disallow AI Training, which blocks the training-only crawlers run by companies such as OpenAI and Anthropic across the domain, and Agent blocked on pages with ads, while Search stays allowed; a site without ads gets everything allowed, and either preset can be changed during onboarding.[^11] For those domains, the answer to "may this AI crawler fetch the page?" is given before the request reaches your robots.txt at all. Two things follow from that. **Your robots.txt may not be the operative control any more.** If your site sits behind a CDN that enforces crawler policy, the CDN decides and the file at your document root is a statement of intent. Check what your CDN is configured to do before assuming robots.txt is the whole story. **Blocking training crawlers can block Googlebot.** Googlebot, Bingbot and Applebot all crawl for both search indexing and AI training. Cloudflare's July announcement said that from 15 September these multi-purpose crawlers "will be allowed/blocked according to all of their behaviors".[^6][^7] On that date its Block and Block on pages with ads settings, which "previously did not apply to mixed-use crawlers", began to apply to them, so choosing Block on the Training category now stops Googlebot, search included.[^11] It is a network-level block rather than a robots.txt line, which makes it harder to spot and harder for the crawler to work around, and it fails in an expensive direction: deindexing. **The setting that avoids it is "Disallow AI Training"**, added the same day for these mixed-use crawlers.[^11] It publishes a no-training line in robots.txt and keeps them crawling for search, because Google, Microsoft and Apple have each stated that opting out of training does not affect search ranking. Bing is the exception for now: Microsoft's robots.txt support for a no-training preference is targeted for early 2027, and until it launches the setting does not convey the opt-out to Bing. Bing's own route today is a `noarchive` robots meta tag, which Bing documents as keeping content out of training for Microsoft's generative AI models but also as "Do not link in Chat and Copilot"; `nocache` limits both uses to the URL, title and snippet.[^12] Any site that already had Training set to Block or Block on pages with ads, whether through the legacy Block AI Bots preset or the granular per-category controls, was migrated to Disallow AI Training automatically, so no existing site lost Googlebot on the day. The new-domain preset described above is not retroactive: Cloudflare says existing settings "carry over on their own". If you are behind Cloudflare and rely on Googlebot, check your Security and AI Crawl Control settings for "Disallow AI Training" specifically, rather than assuming a plain "Block" on the Training category is still the safe choice. ## What is the Content-Signal directive? Cloudflare's Content Signals policy adds a directive to robots.txt that declares how content may be *used*, rather than which paths may be fetched: ``` Content-Signal: search=yes, ai-train=no ``` Its standing is unsettled, and no crawler operator has said it acts on it. Google's John Mueller has said as much directly: "It was made up by a CDN, afaik it has no effects whatsoever for any crawler or llm."[^8] No crawler has been shown to honour it and no controlled test has demonstrated an effect. Cloudflare adds it to robots.txt on sites it serves, so you may find it in your own file without having put it there. The cost of carrying it is a few lines; the demonstrated benefit is currently nil. Treat it as a declaration of preference with no enforcement behind it, in the same category as llms.txt, and do not mistake its presence in your file for protection. ## Can crawler identity be enforced? Not through robots.txt, which has no way to distinguish the real Googlebot from anything that claims to be it. The work aimed at that problem is cryptographic rather than declarative. Web Bot Auth has bots sign their requests with a verifiable key instead of asserting an identity in a header,[^10] and Private Access Control Tokens (PACT), announced by Cloudflare on 22 June 2026 with Mozilla, Google, Microsoft and Shopify as a protocol to be submitted for standardisation, applies the same principle to separating legitimate traffic from abuse.[^17] Neither is something you implement in robots.txt today, and neither changes what you should write in that file. They matter because they show where the enforcement problem is being solved: at the network layer, with signatures, rather than in a text file that asks politely. ## Auditing crawlability The basic audit: 1. **Read your live robots.txt.** Check it is what you expect. Verify nothing critical is disallowed. 2. **Use the Google Search Console URL Inspection tool.** Spot-check key URLs for crawl status, indexing status, and any blocking signals. 3. **Crawl your own site.** Tools like Screaming Frog or Sitebulb can crawl with Googlebot's user agent and report blocked URLs, redirect chains, broken links, and orphan pages. 4. **Review the Crawl Stats report in Search Console.** Track crawl rate, response times, and host status over time. Sudden changes warrant investigation. For a full crawlability checklist, see the [Technical SEO Audit Checklist](/resources/technical-seo-audit-checklist/). ## Frequently asked questions ### Is robots.txt case-sensitive? Yes. `Disallow: /Admin/` and `Disallow: /admin/` block different URLs. ### Does robots.txt affect rankings? Indirectly. By controlling what gets crawled and how crawl budget is spent, it influences indexing efficiency and the freshness of indexed content. The file itself is not a ranking signal. ### Should I disallow the search results pages on my site? Generally yes. Internal search results pages are typically low-value, near-infinite in URL combinations, and should not appear in Google's index. The reliable sequence is to `noindex` them on-page first, keeping them crawlable so Googlebot can read the directive and drop them from the index. Once they have been deindexed, you can add a `robots.txt` disallow to save crawl budget. Do not apply both at once: a disallow on its own cannot deindex a page, because Googlebot never reaches the `noindex`. [^1]: [From Googlebot to GPTBot: who's crawling your site in 2025 — Cloudflare](https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/) [^2]: [robots.txt specification — Google Search Central](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) [^3]: [Spot robots.txt violations in Bot Analytics — Microsoft Clarity Blog](https://clarity.microsoft.com/blog/robots-txt-violations-in-bot-analytics/) [^4]: [Manage how your site appears in Search generative AI features — Google Search Help](https://support.google.com/webmasters/answer/16908024). Carries Google's note: "As of August 31, 2026, we've rolled out this control to all websites worldwide." [^5]: [OpenAI crawlers — OpenAI developer documentation](https://developers.openai.com/api/docs/bots) [^6]: [Content Independence Day: bringing AI options to every site — Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/) [^7]: [Cloudflare's AI crawler rules can block Googlebot — Search Engine Journal](https://www.searchenginejournal.com/cloudflares-ai-crawler-rules-can-block-googlebot/581385/) [^8]: [John Mueller's reply in r/TechSEO — Reddit](https://www.reddit.com/r/TechSEO/comments/1uo1rpb/comment/ovufj15/), 6 July 2026; reported by [Search Engine Roundtable](https://www.seroundtable.com/google-cloudflare-content-signals-41631.html) [^9]: [About Applebot — Apple Support](https://support.apple.com/en-us/119829) [^10]: [Forget IPs: using cryptography to verify bot and agent traffic — Cloudflare](https://blog.cloudflare.com/web-bot-auth/) [^11]: [Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) [^12]: [Robots meta tags and attributes that Bing supports — Bing Webmaster Tools](https://www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240) [^13]: [Agentic Resource Discovery Specification v0.91, agenticresourcediscovery.org](https://agenticresourcediscovery.org/spec/), section 5.1 on the well-known URI and its predecessor path. [^14]: [Optimize your crawl budget — Google Crawling documentation](https://developers.google.com/crawling/docs/crawl-budget) [^15]: [Google-Extended — Google Crawling documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended) [^16]: [Perplexity Crawlers — Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) [^17]: [Cloudflare collaborates with leading browsers to develop a privacy-first protocol for the global Internet — Cloudflare press release](https://www.cloudflare.com/press/press-releases/2026/cloudflare-collaborates-with-leading-browsers-to-develop-a-privacy-first-protocol-for-the-global-internet/), 22 June 2026