Reference

robots.txt Reference

Last updated 15 September 2026 View as Markdown

robots.txt is a plain text file placed at the root of a domain (https://example.com/robots.txt) that tells compliant crawlers which URLs they may and may not fetch. It is part of the Robots Exclusion Protocol, a convention rather than an enforced standard: well-behaved crawlers follow it, malicious ones do not.

For the SEO implications of crawl control, see Crawlability and robots.txt.

File location and format

The file must be at the root of the domain. It applies only to the domain it is hosted on: a robots.txt at example.com does not apply to subdomain.example.com, and vice versa.

Requirements:

Plain text, UTF-8 encoded
Under 500 KiB: Google ignores anything past that limit, so consolidate rules on a large file¹
One directive per line
Lines beginning with # are comments and are ignored by crawlers
Blank lines separate rule groups

Directives

User-agent

Specifies which crawler the following rules apply to. Must appear at the start of each rule group, before any Disallow or Allow directives.

User-agent: Googlebot
User-agent: *

* matches all crawlers not covered by a specific rule group
Multiple User-agent lines can precede a shared set of rules
A crawler obeys exactly one group: the one whose User-agent line most specifically matches its name. Every other group is ignored
A named group does not combine with the * group. If a crawler has its own group, the * group is never consulted for it. See Group selection below
Two groups naming the same crawler are different: Google merges those internally, so splitting one bot’s rules across the file does not lose any of them¹

Disallow

Blocks the specified path. The crawler will not fetch any URL beginning with this path.

Disallow: /admin/
Disallow: /search
Disallow: /

An empty Disallow: value means “allow everything” (equivalent to no restriction)
Disallow: / blocks the entire site
Paths are case-sensitive: Disallow: /Admin/ and Disallow: /admin/ are different rules
The rule matches the beginning of the URL path: Disallow: /blog matches /blog, /blog/, and /blog-post

Allow

Explicitly permits a path within a disallowed directory. Used to create exceptions.

Disallow: /private/
Allow: /private/public-page/

Only meaningful within a Disallow context; Allow: / with no Disallow does nothing
When both Allow and Disallow rules match a URL, the more specific (longer) rule wins
If rules are the same length, Allow wins

Sitemap

Declares the location of an XML sitemap. Not a crawl rule: it is a hint to crawlers about where to find URLs.

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Use absolute URLs, not relative paths
Multiple Sitemap directives are allowed
Can appear anywhere in the file, but convention places them at the end
Googlebot also discovers sitemaps submitted via Search Console; this directive covers other crawlers

Crawl-delay

Specifies a delay (in seconds) between successive requests from the crawler.

Crawl-delay: 2

Googlebot does not honour Crawl-delay. Googlebot now manages its own crawl rate automatically, backing off when the server is slow or returns errors. The manual crawl rate setting in Search Console was removed in January 2024; use Google’s Googlebot report form to flag unusual crawl activity.
Bing, Yandex, and many other crawlers do honour it
Useful on low-resource servers to prevent crawlers from overwhelming the host

Wildcards

* (asterisk)

Matches any sequence of characters, including none.

Disallow: /*.pdf$
Disallow: /search?*
Disallow: /*/print/

Supported by Googlebot; behaviour varies across other crawlers
Disallow: /search?* blocks all URLs containing /search? followed by anything
Disallow: /*/print/ blocks any path with /print/ as a segment

$ (end of string)

Anchors the pattern to the end of the URL.

Disallow: /*.pdf$

Matches only URLs ending with the specified pattern
Disallow: /*.pdf$ blocks /document.pdf but not /pdf-guide/
Supported by Googlebot; behaviour varies across other crawlers
Without $, the pattern matches anywhere in the URL

Group selection

Group selection and rule precedence are two different things, and conflating them causes a dangerous robots.txt mistake.

Group selection happens first. A crawler scans the file, finds the group whose User-agent line most specifically matches its own name, and obeys only that group. Every other group is ignored, including User-agent: *. Google’s spec states it plainly: “User agent specific groups and global groups (*) are not combined.”¹

The * group is not a base layer that named groups build on. It is a fallback for crawlers with no group of their own. So a named group does not “win conflicts” against the * group, it replaces it wholesale. Non-conflicting Disallow rules in the * group are never even read for a crawler that has its own group.

Example:

User-agent: *
Disallow: /private/
Disallow: /cart/

User-agent: GPTBot
Disallow: /no-ai/

GPTBot matches its own group and follows only Disallow: /no-ai/. It never consults the * group, so /private/ and /cart/ are open to it. This fails open: the URLs you meant to block are silently exposed, with no error to warn you.

To apply shared rules to a named crawler, repeat them inside that crawler’s group:

User-agent: *
Disallow: /private/
Disallow: /cart/

User-agent: GPTBot
Disallow: /private/
Disallow: /cart/
Disallow: /no-ai/

This scales badly across many named bots, so generate the file from a template rather than maintaining each group by hand.

User agent specific groups and global groups (*) are not combined

The wildcard group (*) is a fallback, not a global rule. The moment a crawler has its own named group, the * group stops applying to it completely. If you want a shared rule to bind a named bot, it has to be written inside that bot's group. Nothing is inherited.

Rule precedence

Rule precedence applies within the single group a crawler has already selected. When several lines in that group match a URL, crawlers resolve them in this order:

The most specific (longest) matching rule wins
If two rules are equal length, Allow wins over Disallow

Example:

User-agent: *
Disallow: /private/
Allow: /private/public/

A request to /private/public/page matches both rules. /private/public/ (length 16) is longer than /private/ (length 9), so Allow wins and the URL is accessible.

Common user agent strings

Google

User agent	Crawls
`Googlebot`	Web search (applies to all Googlebot variants unless overridden)
`Googlebot-Image`	Google Images
`Googlebot-Video`	Google Video
`Googlebot-News`	Google News
`Google-Extended`	Google AI (Gemini training and products)
`AdsBot-Google`	Google Ads landing page quality
`Mediapartners-Google`	AdSense

Other search engines

User agent	Crawls
`Bingbot`	Bing web search
`Slurp`	Yahoo (uses Bing index)
`DuckDuckBot`	DuckDuckGo
`Baiduspider`	Baidu
`YandexBot`	Yandex

AI crawlers

User agent	Purpose
`GPTBot`	OpenAI training crawler
`OAI-SearchBot`	OpenAI ChatGPT Search indexing
`ChatGPT-User`	OpenAI live retrieval at query time
`ClaudeBot`	Anthropic training crawler
`Claude-SearchBot`	Anthropic search indexing
`Claude-User`	Anthropic live retrieval at a user’s request
`PerplexityBot`	Perplexity indexing crawler
`Perplexity-User`	Perplexity live retrieval
`CCBot`	Common Crawl (used by many model providers)
`Meta-ExternalAgent`	Meta training and direct content indexing
`Meta-WebIndexer`	Meta AI search indexing
`Meta-ExternalFetcher`	Meta live retrieval at a user’s request
`Meta-ExternalAds`	Meta advertising and business products
`FacebookExternalHit`	Meta share-preview metadata

Four of these are exempted from robots.txt by their own operators, and all four are the ones that fetch a page because a person asked for it: ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and FacebookExternalHit. Blocking them takes server-side access controls, not a directive in this file. Claude-User is the exception among the live fetchers: Anthropic documents robots.txt control for it.² Tokens that circulate in bot directories but not in any operator’s documentation, such as anthropic-ai or cohere-ai, are not worth a rule: Anthropic lists only the three agents above, and Cohere states that it does not use bots to crawl the web for training at all.³ See AI Crawler User-Agents for each operator’s documented position.

Common configurations

Block all crawlers

User-agent: *
Disallow: /

Use during development. Remove before launch: this is a common cause of sites not appearing in search after deployment.

Block all AI training crawlers

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

This blocks training only. The search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the live fetchers (ChatGPT-User, Claude-User, Perplexity-User) are deliberately left out, so AI search products can still retrieve and cite the content. Note that Meta-ExternalAgent is documented for both training and content indexing, so blocking it affects Meta’s AI search too. Training and retrieval use different bots; add a search crawler to this list only if you also want to stop being cited.

Your CDN may be answering this question for you

robots.txt is a request, and it is no longer the only layer deciding whether an AI crawler reaches the site. Bot management at the CDN can block a crawler that robots.txt allows, and the block happens before the request ever reaches your server, so nothing in your robots.txt file will show it.

Cloudflare now groups AI crawlers into three categories rather than by vendor: training (crawlers taking content to train or fine-tune a model), search (crawlers indexing content to answer questions about it later) and agent (automated activity acting in real time on a person’s behalf).⁴ Since 15 September 2026, a domain newly onboarded to Cloudflare that says it earns from ads is offered a preset with Training set to Disallow AI Training, which blocks training-only crawlers, and Agent blocked on pages that display ads, while Search remains allowed.⁴⁵

Two things worth being precise about, because they are easy to conflate. The preset is narrow: it applies to newly onboarded domains, and only to sites that say they carry ads. A new editorial or business site with no ad units gets everything allowed, and Search crawlers stay allowed either way, so citations drawn from a search index are not touched by the preset. On pages with ads it does block the Agent category, which covers chat assistants fetching a page live on a person’s behalf.⁵

The wider risk is one you opt into. Googlebot, Applebot and Bingbot crawl for both search and training, and Cloudflare’s Block setting applies to them, so if you choose Block on the Training category, that also blocks all three.⁵ A setting picked to keep content out of model training can therefore take the site out of Google. The setting that avoids it is Disallow AI Training, which publishes a no-training line in robots.txt and keeps them crawling for search; Bing will not receive that opt-out until Microsoft’s robots.txt support arrives, targeted for early 2027.⁵ Block is not the default; it is a decision, and it is the one to get right. If the site is behind Cloudflare, read the security settings rather than assuming robots.txt is the whole picture.

Block internal search results

User-agent: *
Disallow: /search
Disallow: /search/

Block admin and account areas

User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /login
Disallow: /checkout/

Block parameterised URLs

User-agent: *
Disallow: /*?

Blocks all parameterised URLs. Use with caution: this can inadvertently block legitimate pages if any use query strings.

Block print and feed versions

User-agent: *
Disallow: /*/print/
Disallow: /*/feed/
Disallow: /*/amp/

Allow Googlebot, block all others

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

An empty Disallow after a specific user agent allows full access for that crawler. Useful for staging environments where you want Google to crawl but no other bots.

What robots.txt does not do

It does not prevent indexing. A URL blocked in robots.txt can still appear in search results if it has inbound links. Google will index the URL without crawling it, showing the page title from anchor text and no description. To prevent indexing, use a noindex meta tag, but the page must be crawlable for Google to read it.

It does not secure content. robots.txt is public and explicitly advertises what you are hiding. Do not rely on it for sensitive content; use authentication.

It does not apply to subdomains. example.com/robots.txt has no authority over blog.example.com. Each subdomain needs its own file.

It applies to HTTP, HTTPS and, for Google, FTP. Google’s spec lists all three as supported protocols, each with its own file; nothing else is covered.¹

It does not stop malicious bots. Only well-behaved crawlers follow robots.txt. Scrapers, spam bots, and vulnerability scanners ignore it.

robots.txt specification — Google Search Central ↩ ↩² ↩³ ↩⁴
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic. Lists ClaudeBot, Claude-User and Claude-SearchBot, and states that opting out “requires modifying the robots.txt file”. ↩
Cohere Web Crawlers — Cohere. “We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train” its models. ↩
Your site, your rules: new AI traffic options for all customers — Cloudflare ↩ ↩²
Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog, 15 September 2026 ↩ ↩² ↩³ ↩⁴