--- title: "robots.txt Reference" description: "Complete robots.txt syntax reference: every directive, wildcard pattern, user agent string, and common configuration example for controlling crawler access." tldr: "robots.txt uses four main directives: User-agent (which crawler the rules apply to), Disallow (paths to block), Allow (paths to permit within a disallowed directory), and Sitemap (where to find your XML sitemap). The file does not prevent indexing, only crawling. A disallowed page can still appear in search results if it has inbound links. Malicious bots ignore it entirely. Wildcards (* and $) extend pattern matching but have inconsistencies between crawlers. And robots.txt is not the only layer deciding crawler access: a CDN or bot-management rule can block a crawler your file allows, before the request reaches your server, and nothing in the file will show it." publishDate: 2026-04-27 updatedDate: 2026-10-19 author: "Liam Hayward" --- robots.txt is a plain text file placed at the root of a domain (`https://example.com/robots.txt`) that tells compliant crawlers which URLs they may and may not fetch. It is part of the Robots Exclusion Protocol, a convention rather than an enforced standard: well-behaved crawlers follow it, malicious ones do not. For the SEO implications of crawl control, see [Crawlability and robots.txt](/technical-seo/crawlability-and-robots-txt/). --- ## File location and format The file must be at the root of the domain. It applies only to the domain it is hosted on: a robots.txt at `example.com` does not apply to `subdomain.example.com`, and vice versa. Requirements: - Plain text, UTF-8 encoded - Under 500 KiB: Google ignores anything past that limit, so consolidate rules on a large file[^1] - One directive per line - Lines beginning with `#` are comments and are ignored by crawlers - Blank lines separate rule groups --- ## Directives ### User-agent Specifies which crawler the following rules apply to. Must appear at the start of each rule group, before any `Disallow` or `Allow` directives. ``` User-agent: Googlebot User-agent: * ``` - `*` matches all crawlers not covered by a specific rule group - Multiple `User-agent` lines can precede a shared set of rules - A crawler obeys exactly one group: the one whose `User-agent` line most specifically matches its name. Every other group is ignored - A named group does **not** combine with the `*` group. If a crawler has its own group, the `*` group is never consulted for it. See [Group selection](#group-selection) below - Two groups naming the *same* crawler are different: Google merges those internally, so splitting one bot's rules across the file does not lose any of them[^1] ### Disallow Blocks the specified path. The crawler will not fetch any URL beginning with this path. ``` Disallow: /admin/ Disallow: /search Disallow: / ``` - An empty `Disallow:` value means "allow everything" (equivalent to no restriction) - `Disallow: /` blocks the entire site - Paths are case-sensitive: `Disallow: /Admin/` and `Disallow: /admin/` are different rules - The rule matches the beginning of the URL path: `Disallow: /blog` matches `/blog`, `/blog/`, and `/blog-post` ### Allow Explicitly permits a path within a disallowed directory. Used to create exceptions. ``` Disallow: /private/ Allow: /private/public-page/ ``` - Only meaningful within a `Disallow` context; `Allow: /` with no `Disallow` does nothing - When both `Allow` and `Disallow` rules match a URL, the more specific (longer) rule wins - If rules are the same length, `Allow` wins ### Sitemap Declares the location of an XML sitemap. Not a crawl rule: it is a hint to crawlers about where to find URLs. ``` Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-news.xml ``` - Use absolute URLs, not relative paths - Multiple `Sitemap` directives are allowed - Can appear anywhere in the file, but convention places them at the end - Googlebot also discovers sitemaps submitted via Search Console; this directive covers other crawlers ### Crawl-delay Specifies a delay (in seconds) between successive requests from the crawler. ``` Crawl-delay: 2 ``` - **Googlebot does not honour `Crawl-delay`**. Googlebot now manages its own crawl rate automatically, backing off when the server is slow or returns errors. The manual crawl rate setting in Search Console was removed in January 2024; use Google's Googlebot report form to flag unusual crawl activity. - Bing, Yandex, and many other crawlers do honour it - Useful on low-resource servers to prevent crawlers from overwhelming the host --- ## Wildcards ### * (asterisk) Matches any sequence of characters, including none. ``` Disallow: /*.pdf$ Disallow: /search?* Disallow: /*/print/ ``` - Supported by Googlebot; behaviour varies across other crawlers - `Disallow: /search?*` blocks all URLs containing `/search?` followed by anything - `Disallow: /*/print/` blocks any path with `/print/` as a segment ### $ (end of string) Anchors the pattern to the end of the URL. ``` Disallow: /*.pdf$ ``` - Matches only URLs ending with the specified pattern - `Disallow: /*.pdf$` blocks `/document.pdf` but not `/pdf-guide/` - Supported by Googlebot; behaviour varies across other crawlers - Without `$`, the pattern matches anywhere in the URL --- ## Group selection Group selection and rule precedence are two different things, and conflating them causes a dangerous robots.txt mistake. Group selection happens first. A crawler scans the file, finds the group whose `User-agent` line most specifically matches its own name, and obeys **only that group**. Every other group is ignored, including `User-agent: *`. Google's spec states it plainly: "User agent specific groups and global groups (`*`) are not combined."[^1] The `*` group is not a base layer that named groups build on. It is a fallback for crawlers with no group of their own. So a named group does not "win conflicts" against the `*` group, it **replaces it wholesale**. Non-conflicting `Disallow` rules in the `*` group are never even read for a crawler that has its own group. **Example:** ``` User-agent: * Disallow: /private/ Disallow: /cart/ User-agent: GPTBot Disallow: /no-ai/ ``` GPTBot matches its own group and follows only `Disallow: /no-ai/`. It never consults the `*` group, so `/private/` and `/cart/` are **open to it**. This fails open: the URLs you meant to block are silently exposed, with no error to warn you. To apply shared rules to a named crawler, repeat them inside that crawler's group: ``` User-agent: * Disallow: /private/ Disallow: /cart/ User-agent: GPTBot Disallow: /private/ Disallow: /cart/ Disallow: /no-ai/ ``` This scales badly across many named bots, so generate the file from a template rather than maintaining each group by hand.

User agent specific groups and global groups (*) are not combined

The wildcard group (*) is a fallback, not a global rule. The moment a crawler has its own named group, the * group stops applying to it completely. If you want a shared rule to bind a named bot, it has to be written inside that bot's group. Nothing is inherited.

--- ## Rule precedence Rule precedence applies **within the single group a crawler has already selected**. When several lines in that group match a URL, crawlers resolve them in this order: 1. The most specific (longest) matching rule wins 2. If two rules are equal length, `Allow` wins over `Disallow` **Example:** ``` User-agent: * Disallow: /private/ Allow: /private/public/ ``` A request to `/private/public/page` matches both rules. `/private/public/` (length 16) is longer than `/private/` (length 9), so `Allow` wins and the URL is accessible. --- ## Common user agent strings ### Google | User agent | Crawls | |-----------|--------| | `Googlebot` | Web search (applies to all Googlebot variants unless overridden) | | `Googlebot-Image` | Google Images | | `Googlebot-Video` | Google Video | | `Googlebot-News` | Google News | | `Google-Extended` | Nothing: a control token for Gemini training and grounding, fetched by Google's existing crawlers | | `AdsBot-Google` | Google Ads landing page quality | | `Mediapartners-Google` | AdSense | ### Other search engines | User agent | Crawls | |-----------|--------| | `Bingbot` | Bing web search | | `Slurp` | Yahoo (uses Bing index) | | `DuckDuckBot` | DuckDuckGo | | `Baiduspider` | Baidu | | `YandexBot` | Yandex | ### AI crawlers | User agent | Purpose | |-----------|---------| | `GPTBot` | OpenAI training crawler | | `OAI-SearchBot` | OpenAI ChatGPT Search indexing | | `ChatGPT-User` | OpenAI live retrieval at query time | | `ClaudeBot` | Anthropic training crawler | | `Claude-SearchBot` | Anthropic search indexing | | `Claude-User` | Anthropic live retrieval at a user's request | | `PerplexityBot` | Perplexity indexing crawler | | `Perplexity-User` | Perplexity live retrieval | | `CCBot` | Common Crawl (used by many model providers) | | `Meta-ExternalAgent` | Meta training and direct content indexing | | `Meta-WebIndexer` | Meta AI search indexing | | `Meta-ExternalFetcher` | Meta live retrieval at a user's request | | `Meta-ExternalAds` | Meta advertising and business products | | `FacebookExternalHit` | Meta share-preview metadata | Four of these are exempted from robots.txt by their own operators, and all four are the ones that fetch a page because a person asked for it: `ChatGPT-User`, `Perplexity-User`, `Meta-ExternalFetcher` and `FacebookExternalHit`. Blocking them takes server-side access controls, not a directive in this file. `Claude-User` is the exception among the live fetchers: Anthropic documents robots.txt control for it.[^3] Tokens that circulate in bot directories but not in any operator's documentation, such as `anthropic-ai` or `cohere-ai`, are not worth a rule: Anthropic lists only the three agents above, and Cohere states that it does not use bots to crawl the web for training at all.[^4] See AI Crawler User-Agents for each operator's documented position. --- ## Common configurations ### Block all crawlers ``` User-agent: * Disallow: / ``` Use during development. Remove before launch: this is a common cause of sites not appearing in search after deployment. ### Block all AI training crawlers ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / ``` This blocks training only. The search crawlers (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) and the live fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) are deliberately left out, so AI search products can still retrieve and cite the content. Note that `Meta-ExternalAgent` is documented for both training and content indexing, so blocking it affects Meta's AI search too. Training and retrieval use different bots; add a search crawler to this list only if you also want to stop being cited. ### Your CDN may be answering this question for you robots.txt is a request, and it is no longer the only layer deciding whether an AI crawler reaches the site. Bot management at the CDN can block a crawler that robots.txt allows, and the block happens before the request ever reaches your server, so nothing in your robots.txt file will show it. Cloudflare now groups AI crawlers into three categories rather than by vendor: **training** (crawlers taking content to train or fine-tune a model), **search** (crawlers indexing content to answer questions about it later) and **agent** (automated activity acting in real time on a person's behalf).[^2] Since 15 September 2026, a domain newly onboarded to Cloudflare that says it earns from ads is offered a preset with Training set to Disallow AI Training, which blocks training-only crawlers, and Agent blocked on pages that display ads, while Search remains allowed.[^2][^5] Two things worth being precise about, because they are easy to conflate. The preset is narrow: it applies to newly onboarded domains, and only to sites that say they carry ads. A new editorial or business site with no ad units gets everything allowed, and Search crawlers stay allowed either way, so citations drawn from a search index are not touched by the preset. On pages with ads it does block the Agent category, which covers chat assistants fetching a page live on a person's behalf.[^5] The wider risk is one you opt into. Googlebot, Applebot and Bingbot crawl for both search and training, and Cloudflare's Block setting applies to them, so if you *choose* Block on the Training category, that also blocks all three.[^5] A setting picked to keep content out of model training can therefore take the site out of Google. The setting that avoids it is Disallow AI Training, which publishes a no-training line in robots.txt and keeps them crawling for search; Bing will not receive that opt-out until Microsoft's robots.txt support arrives, targeted for early 2027.[^5] Block is not the default; it is a decision, and it is the one to get right. If the site is behind Cloudflare, read the security settings rather than assuming robots.txt is the whole picture. ### Block internal search results ``` User-agent: * Disallow: /search Disallow: /search/ ``` ### Block admin and account areas ``` User-agent: * Disallow: /admin/ Disallow: /account/ Disallow: /login Disallow: /checkout/ ``` ### Block parameterised URLs ``` User-agent: * Disallow: /*? ``` Blocks all parameterised URLs. Use with caution: this can inadvertently block legitimate pages if any use query strings. ### Block print and feed versions ``` User-agent: * Disallow: /*/print/ Disallow: /*/feed/ Disallow: /*/amp/ ``` ### Allow Googlebot, block all others ``` User-agent: * Disallow: / User-agent: Googlebot Disallow: ``` An empty `Disallow` after a specific user agent allows full access for that crawler. Useful for staging environments where you want Google to crawl but no other bots. --- ## What robots.txt does not do **It does not prevent indexing.** A URL blocked in robots.txt can still appear in search results if it has inbound links. Google will index the URL without crawling it, showing the page title from anchor text and no description. To prevent indexing, use a `noindex` meta tag, but the page must be crawlable for Google to read it. **It does not secure content.** robots.txt is public and explicitly advertises what you are hiding. Do not rely on it for sensitive content; use authentication. **It does not apply to subdomains.** `example.com/robots.txt` has no authority over `blog.example.com`. Each subdomain needs its own file. **It applies to HTTP, HTTPS and, for Google, FTP.** Google's spec lists all three as supported protocols, each with its own file; nothing else is covered.[^1] **It does not stop malicious bots.** Only well-behaved crawlers follow robots.txt. Scrapers, spam bots, and vulnerability scanners ignore it. [^1]: [robots.txt specification — Google Search Central](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) [^2]: [Your site, your rules: new AI traffic options for all customers — Cloudflare](https://blog.cloudflare.com/content-independence-day-ai-options/) [^3]: [Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler). Lists ClaudeBot, Claude-User and Claude-SearchBot, and states that opting out "requires modifying the robots.txt file". [^4]: [Cohere Web Crawlers — Cohere](https://docs.cohere.com/docs/cohere-web-crawlers). "We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train" its models. [^5]: [Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), 15 September 2026