--- title: "Google, Microsoft, OpenAI and Anthropic Back One robots.txt Opt-Out for AI Training" description: "One robots.txt group will opt a site out of AI training at Google, Microsoft, OpenAI and Anthropic from 2027. Bing also retires noarchive as an AI control." tldr: "The shared AI training opt-out is a single robots.txt group, addressed to a shared token instead of a named crawler, that asks every participating AI company not to train models on a site's content without blocking any crawler. Google, Microsoft, OpenAI and Anthropic announced support on 30 October 2026, based on an IETF draft written at Anthropic. None honours it yet: Microsoft starts on 1 February 2027, Google in early 2027. At Google it counts as disallowing Google-Extended, which also removes the site from grounding in Gemini Apps. Bing also splits training from grounding with new page-level settings, and stops reading noarchive and nocache as AI preferences on 1 March 2027." publishDate: 2026-10-31 author: "Liam Hayward" --- On 30 October 2026 Google, Microsoft, OpenAI and Anthropic each announced that they will honour one robots.txt signal for AI training. A site that adds a single group opts out of training at every company that supports it, instead of naming each company's crawler in turn. The syntax comes from an Internet-Draft, a working document submitted to the IETF (the body that standardises internet protocols), written by Kevin Kelley of Anthropic and published the same day. Microsoft went further than the others: it added a separate setting for whether its AI answers may draw on a page, and set a deadline for sites that rely on its older `noarchive` and `nocache` tags. ## What does the shared AI training signal look like? This example is from the draft: ``` User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ``` It states a preference against AI training for the whole site except `/press-kit/`. The draft calls `__usage_ai-training` a "policy token": it sits where a crawler's name would go, but no crawler sends it. Three rules in the draft decide how it behaves: - **It is not a crawl rule.** The draft says the token "is not a directive for crawling behavior". It states how content may be used, so a crawler that reads it can keep fetching. To stop a crawler fetching pages, you still need a group naming that crawler. - **`User-agent: *` does not match it.** An existing catch-all group says nothing about training under this draft. - **Software that does not implement it ignores the group**, as it would any user-agent it does not recognise, so adding the group cannot change how your existing rules are read. "AI training" takes its definition from the IETF AI Preferences working group's vocabulary draft: "Using an asset to modify the learned parameters of a generative AI model." Training models used only for classification, ranking or scoring falls outside it. Kelley's document is an individual draft proposed to that working group, not an adopted standard, and it expires on 3 May 2027 unless revised. ## How is the shared signal different from Cloudflare's Content Signals? Cloudflare launched Content Signals on 24 September 2025: a `Content-signal:` line in robots.txt with yes or no values for `search`, `ai-input` and `ai-train`. It serves `search=yes, ai-train=no` by default on domains using its managed robots.txt, more than 3.8 million at launch, and submitted the format to the IETF AI Preferences working group as a draft in October 2025. None of the four 30 October announcements mentions it, and in July 2026 John Mueller, Google Search Advocate, said that, as far as he was aware, no crawler or language model acts on it. With Cloudflare's "Disallow AI Training" setting and Bot Preference Sync turned on, the robots.txt Cloudflare manages for a site contains these lines, alongside `Disallow` groups for other training crawlers such as `ClaudeBot` and `CCBot`: ``` User-Agent: * Content-signal: search=yes, ai-train=no, use=reference Allow: / User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / ``` The shared signal expresses the same training preference by a different route. `Content-signal` is a new directive, which software that does not recognise it skips, and which Cloudflare's documentation says Search Console can flag as "Syntax not understood". The shared signal is an ordinary user-agent group: Kelley's draft "introduces no new directive names and no new grammar". A robots.txt carrying `ai-train=no` has not stated the shared preference, and none of the four announcements says it will be read as one. ## When does each company start honouring the shared signal? - **Microsoft:** from 1 February 2027, as Bingbot crawls each page. - **Google:** "in early 2027", according to a note added on 30 October to its common crawlers documentation. The announcement itself says 2027. - **OpenAI:** "in the coming months", with a further update promised when it ships. - **Anthropic:** Kelley wrote on the IETF mailing list that Anthropic "has already started implementing support", that rollout across its services will take "the coming months", and that he expects "complications and edge cases" along the way. All four say their own controls, such as `GPTBot`, `ClaudeBot` and `Google-Extended`, keep working alongside the shared signal. ## How does Google treat the shared opt-out? Google's announcement, posted by Mueller, says that when a site uses the token to opt out of training and does not name `Google-Extended`, Google "plans to interpret that as if the publisher had disallowed Google-Extended". That reaches further than training. Google's crawler documentation says `Google-Extended` governs training of Gemini models and also grounding, the use of pages to support answers at the time a question is asked, in Gemini Apps and in Grounding with Google Search on Vertex AI. The shared opt-out therefore also takes a site out of Gemini Apps answers. The post says sites that "specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps", but allowing `Google-Extended` also allows Gemini training: at Google, training and Gemini Apps grounding remain one choice. Google Search is unaffected either way, since `Google-Extended` "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". ## What is Bing changing about noarchive and nocache? Bing has read `noarchive` and `nocache` as its AI controls since 2023. Its announcement, by Krishna Madhavan, Principal Product Manager at Microsoft Bing, replaces them with two preferences that can be set independently: - **AI Training**, set for a site or path with the shared robots.txt group, or for one page with `` or the header `X-Robots-Tag: Bingbot: ai-training:disallow`. The page setting overrides robots.txt. - **AI Grounding**, set per page only, with `ai-grounding:disallow` or `ai-grounding:disallowextended` in the same meta tag or header. There is no robots.txt version. `disallowextended` keeps a page available to experiences Microsoft runs itself, which it lists as AI Mode, AI Overview, Copilot and Microsoft 365, and withholds it from the grounding services Microsoft provides to other companies, such as Web IQ and grounding through the Foundry API. Both default to Allow, and neither changes eligibility for Bing search results. Microsoft warns that choosing Disallow or DisallowExtended for grounding "will significantly reduce your content's visibility in AI-generated experiences". For sites already using `noarchive` or `nocache`, the transition runs on four dates: - **15 December 2026:** a temporary grounding preference opens in Bing Webmaster Tools, available only to sites using `noarchive` or `nocache` on that date. - **1 February 2027:** Bing starts reading the new preferences as it crawls. - **1 March 2027:** `noarchive` and `nocache` stop counting as training or grounding preferences. A page that still relies on them, with no temporary preference recorded, defaults to Allow for grounding until Bing recrawls it and finds a new tag. - **30 September 2027:** temporary preferences expire. The temporary preference exists because a tag published in time may not be read in time: Microsoft says it may not recrawl a site between February and 1 March. ## What the shared training opt-out means for your robots.txt **If you block AI training today,** keep your existing groups for `GPTBot`, `ClaudeBot`, `Google-Extended` and `CCBot`. Nothing honours the shared token yet, and crawlers run by companies that have not announced support will ignore it. Adding the group now costs one robots.txt edit and changes nothing for parsers that do not read it. **If you are behind Cloudflare and use its "Disallow AI Training" setting,** add the shared group to your own robots.txt if you want Bing to receive the opt-out. As documented on 31 October 2026, the file Cloudflare manages for that setting, shown above, has no `__usage_ai-training` group and nothing for Bing, which the setting [has never reached](/seo-news-updates/cloudflare-training-opt-out-protects-googlebot/). **If you want to appear in Gemini Apps answers,** the shared opt-out removes you there unless you name `Google-Extended`, and naming it with Allow lets Google train on your content. Decide Google separately. **If you use `noarchive` or `nocache` for Bing,** act before 1 March 2027: publish your training preference (the shared group or the page-level `ai-training` tag), and for grounding add `ai-grounding` tags to your pages and record the temporary preference in Bing Webmaster Tools once it opens on 15 December. If you used `noarchive` only to keep content out of training, the split lets you keep that opt-out while becoming eligible for Copilot citations again, which `noarchive` blocked: disallow training and leave grounding at its default. **If you have set none of these,** nothing changes. Every preference described here defaults to Allow. ## Sources - [Supporting a shared AI training control for websites, Google Search Central Blog, 30 October 2026](https://developers.google.com/search/blog/2026/10/ai-training-control) - [Google-Extended, Google's common crawlers, Google Crawling Infrastructure](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended) - [Your content, your choice: clearer preferences for training and grounding, Microsoft Bing Blog, 30 October 2026](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) - [Making it easier for websites to express AI training preferences, OpenAI, 30 October 2026](https://openai.com/index/helping-websites-express-ai-training-preferences/) - [New draft: draft-kelley-aipref-training-preference-signal-00, Kevin Kelley, IETF ai-control mailing list, 30 October 2026](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) - [Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00, IETF](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) - [robots.txt setting, Cloudflare Bots documentation](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) - [Giving users choice with Cloudflare's new Content Signals Policy, Cloudflare Blog, 24 September 2025](https://blog.cloudflare.com/content-signals-policy/) - [John Mueller on Content Signals, r/TechSEO (Reddit), 6 July 2026](https://www.reddit.com/r/TechSEO/comments/1uo1rpb/comment/ovufj15/)