Industry

Google, Microsoft, OpenAI and Anthropic Back One robots.txt Opt-Out for AI Training

Markdown
View as Markdown Open this page as plain text
This page as Markdown, ready to paste into ChatGPT, Claude or a doc.
RSS
Two lines of robots.txt code in large monospace type fill a cream background: User-Agent: __usage_ai-training, then Disallow: /. On the right, a round silver robot with a star-tipped antenna walks away frowning, holding a blank notepad in one hand and a pencil in the other.
One group in robots.txt, and a training bot leaving with a blank page. Illustration: AI-generated.

On 30 October 2026 Google, Microsoft, OpenAI and Anthropic each announced that they will honour one robots.txt signal for AI training. A site that adds a single group opts out of training at every company that supports it, instead of naming each company’s crawler in turn. The syntax comes from an Internet-Draft, a working document submitted to the IETF (the body that standardises internet protocols), written by Kevin Kelley of Anthropic and published the same day. Microsoft went further than the others: it added a separate setting for whether its AI answers may draw on a page, and set a deadline for sites that rely on its older noarchive and nocache tags.

What does the shared AI training signal look like?

This example is from the draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

It states a preference against AI training for the whole site except /press-kit/. The draft calls __usage_ai-training a “policy token”: it sits where a crawler’s name would go, but no crawler sends it. Three rules in the draft decide how it behaves:

  • It is not a crawl rule. The draft says the token “is not a directive for crawling behavior”. It states how content may be used, so a crawler that reads it can keep fetching. To stop a crawler fetching pages, you still need a group naming that crawler.
  • User-agent: * does not match it. An existing catch-all group says nothing about training under this draft.
  • Software that does not implement it ignores the group, as it would any user-agent it does not recognise, so adding the group cannot change how your existing rules are read.

“AI training” takes its definition from the IETF AI Preferences working group’s vocabulary draft: “Using an asset to modify the learned parameters of a generative AI model.” Training models used only for classification, ranking or scoring falls outside it. Kelley’s document is an individual draft proposed to that working group, not an adopted standard, and it expires on 3 May 2027 unless revised.

How is the shared signal different from Cloudflare’s Content Signals?

Cloudflare launched Content Signals on 24 September 2025: a Content-signal: line in robots.txt with yes or no values for search, ai-input and ai-train. It serves search=yes, ai-train=no by default on domains using its managed robots.txt, more than 3.8 million at launch, and submitted the format to the IETF AI Preferences working group as a draft in October 2025. None of the four 30 October announcements mentions it, and in July 2026 John Mueller, Google Search Advocate, said that, as far as he was aware, no crawler or language model acts on it.

With Cloudflare’s “Disallow AI Training” setting and Bot Preference Sync turned on, the robots.txt Cloudflare manages for a site contains these lines, alongside Disallow groups for other training crawlers such as ClaudeBot and CCBot:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The shared signal expresses the same training preference by a different route. Content-signal is a new directive, which software that does not recognise it skips, and which Cloudflare’s documentation says Search Console can flag as “Syntax not understood”. The shared signal is an ordinary user-agent group: Kelley’s draft “introduces no new directive names and no new grammar”. A robots.txt carrying ai-train=no has not stated the shared preference, and none of the four announcements says it will be read as one.

When does each company start honouring the shared signal?

  • Microsoft: from 1 February 2027, as Bingbot crawls each page.
  • Google: “in early 2027”, according to a note added on 30 October to its common crawlers documentation. The announcement itself says 2027.
  • OpenAI: “in the coming months”, with a further update promised when it ships.
  • Anthropic: Kelley wrote on the IETF mailing list that Anthropic “has already started implementing support”, that rollout across its services will take “the coming months”, and that he expects “complications and edge cases” along the way.

All four say their own controls, such as GPTBot, ClaudeBot and Google-Extended, keep working alongside the shared signal.

How does Google treat the shared opt-out?

Google’s announcement, posted by Mueller, says that when a site uses the token to opt out of training and does not name Google-Extended, Google “plans to interpret that as if the publisher had disallowed Google-Extended”.

That reaches further than training. Google’s crawler documentation says Google-Extended governs training of Gemini models and also grounding, the use of pages to support answers at the time a question is asked, in Gemini Apps and in Grounding with Google Search on Vertex AI. The shared opt-out therefore also takes a site out of Gemini Apps answers. The post says sites that “specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps”, but allowing Google-Extended also allows Gemini training: at Google, training and Gemini Apps grounding remain one choice. Google Search is unaffected either way, since Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”.

What is Bing changing about noarchive and nocache?

Bing has read noarchive and nocache as its AI controls since 2023. Its announcement, by Krishna Madhavan, Principal Product Manager at Microsoft Bing, replaces them with two preferences that can be set independently:

  • AI Training, set for a site or path with the shared robots.txt group, or for one page with <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow. The page setting overrides robots.txt.
  • AI Grounding, set per page only, with ai-grounding:disallow or ai-grounding:disallowextended in the same meta tag or header. There is no robots.txt version. disallowextended keeps a page available to experiences Microsoft runs itself, which it lists as AI Mode, AI Overview, Copilot and Microsoft 365, and withholds it from the grounding services Microsoft provides to other companies, such as Web IQ and grounding through the Foundry API.

Both default to Allow, and neither changes eligibility for Bing search results. Microsoft warns that choosing Disallow or DisallowExtended for grounding “will significantly reduce your content’s visibility in AI-generated experiences”.

For sites already using noarchive or nocache, the transition runs on four dates:

  • 15 December 2026: a temporary grounding preference opens in Bing Webmaster Tools, available only to sites using noarchive or nocache on that date.
  • 1 February 2027: Bing starts reading the new preferences as it crawls.
  • 1 March 2027: noarchive and nocache stop counting as training or grounding preferences. A page that still relies on them, with no temporary preference recorded, defaults to Allow for grounding until Bing recrawls it and finds a new tag.
  • 30 September 2027: temporary preferences expire.

The temporary preference exists because a tag published in time may not be read in time: Microsoft says it may not recrawl a site between February and 1 March.

What the shared training opt-out means for your robots.txt

If you block AI training today, keep your existing groups for GPTBot, ClaudeBot, Google-Extended and CCBot. Nothing honours the shared token yet, and crawlers run by companies that have not announced support will ignore it. Adding the group now costs one robots.txt edit and changes nothing for parsers that do not read it.

If you are behind Cloudflare and use its “Disallow AI Training” setting, add the shared group to your own robots.txt if you want Bing to receive the opt-out. As documented on 31 October 2026, the file Cloudflare manages for that setting, shown above, has no __usage_ai-training group and nothing for Bing, which the setting has never reached.

If you want to appear in Gemini Apps answers, the shared opt-out removes you there unless you name Google-Extended, and naming it with Allow lets Google train on your content. Decide Google separately.

If you use noarchive or nocache for Bing, act before 1 March 2027: publish your training preference (the shared group or the page-level ai-training tag), and for grounding add ai-grounding tags to your pages and record the temporary preference in Bing Webmaster Tools once it opens on 15 December. If you used noarchive only to keep content out of training, the split lets you keep that opt-out while becoming eligible for Copilot citations again, which noarchive blocked: disallow training and leave grounding at its default.

If you have set none of these, nothing changes. Every preference described here defaults to Allow.

Sources

More news

Add as a preferred source on Google