Google, Microsoft, OpenAI and Anthropic will honor a shared robots.txt AI training opt-out so a site can state one preference against generative AI training instead of a rule for every crawler. They announced the plan on October 30, 2026. Most support arrives in 2027, and the signal does not block crawling or ordinary search visibility.
Key Takeaways
- Four AI firms will honor one robots.txt token for generative training opt-outs.
- Google and Microsoft target 2027; OpenAI says the coming months.
- The token does not block crawling or ordinary search visibility.
- Microsoft stops NoArchive and NoCache as training and grounding options after March 1, 2027.
How the Shared AI Training Opt-Out Works
The control is the policy token __usage_ai-training, from an individual Internet Draft Kevin Kelley of Anthropic sent to the IETF AI Preferences working group. It is not a new directive. Sites still use Allow and Disallow. Search Engine Journal reported an example that disallows the whole site and allows /press-kit/.
AI training, in that vocabulary, means using content to modify a generative model’s learned parameters. Non-generative classification and ranking are exempt. A wildcard group never matches the token, and no __usage_ai-training group means no preference. Kelley says that describes every robots.txt file currently published.
The token does not control crawling. The Next Web noted that private agreements can override it. The draft is an individual submission with no formal IETF standing. Search Engine Journal reported that it expires May 3, 2027, unless a new version replaces it.
When Companies Will Honor the Token
Anthropic has started development, and The Verge reported that it has already adopted the signal. Kelley wrote that support across Anthropic’s services will take some time, and that the token would complement existing crawler controls rather than replace them.
The three crawl controls on Anthropic’s crawler-help page, as Search Engine Journal checked it, are ClaudeBot, Claude-User and Claude-SearchBot. Those names come from that page, not from Kelley’s note that the token complements existing controls.
OpenAI aims to implement the signal in the coming months, while GPTBot and its current robots.txt policy stay in effect. Search Engine Journal found no stated rule for a conflict between the shared line and GPTBot.
Google aims to support the control in 2027, and its crawler documentation says early 2027. John Mueller wrote on the Google Search Central blog that if Google-Extended is not set, an opt-out with the token will be read “as if the publisher had disallowed Google-Extended.”
That covers future Gemini training and grounding in Gemini Apps and on Vertex AI, not inclusion in Google Search. AI Overviews and AI Mode keep a separate Search Console opt-out.
Microsoft starts honoring the new preferences on February 1, 2027. It stops recognizing NoArchive and NoCache as training or grounding options after March 1, 2027. Bing principal product manager Krishna Madhavan wrote, “Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.”
| Provider | Support timing | What the token covers |
|---|---|---|
| Anthropic | Already started | Complements existing crawler controls |
| Early 2027 | Google-Extended block if that control is unset | |
| Microsoft | February 1, 2027 | Training only; grounding is separate |
| OpenAI | Coming months | Foundational generative training |
Grounding Is Not the Same Opt-Out
One robots.txt group will not cover the same uses at every company. At Google, training and Gemini grounding both sit under Google-Extended. Crawler documentation describes no way to block Gemini training and still allow Gemini Apps grounding. Allowing Google-Extended keeps a site eligible for Gemini Apps answers and clicks.
Microsoft splits the choice. The shared token covers training. Grounding is page-level, with Allow as the default and Disallow to keep a page out. DisallowExtended allows grounding in Copilot and Bing’s AI Mode but not in grounding services Microsoft sells to other businesses, called Web IQ.
A meta tag or HTTP header can override robots.txt for one page. Microsoft said a grounding block “will significantly reduce your content’s visibility in AI-generated experiences,” without changing Bing results. The new preferences “do not establish licensing or payment arrangements.”
Microsoft wants sites that use NoArchive or NoCache to switch to the new preferences by March 1, 2027. A Bing Webmaster Tools preference opens Dec. 15, only for sites still using those tags that day, if they want to stay out of grounding.
Pages without a preference by March 1 default to allowing grounding until Microsoft recrawls them. After Microsoft sees a new preference, old tags on that site no longer count. Google, OpenAI and Kelley say older controls continue.
What Site Owners Should Do Next
The effort is voluntary, and Meta has not committed. Andy Stone posted support for a common signal on X, The Verge reported, saying, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step.” He did not say Meta will honor the token.
Google said the proposal is consistent with the EU Code of Practice for general-purpose AI under the AI Act. Until each crawler actually reads the token, keep the controls you already rely on.
- Leave Google-Extended, GPTBot and other company rules in place until that provider says the shared token is live.
- If NoArchive or NoCache limits Copilot, switch to the new preferences by March 1, 2027; pages without a preference then default to allowing grounding until recrawl.
- Treat training and grounding as separate choices at Microsoft, and as one Google-Extended block at Google unless you override it.
- Do not treat a robots.txt disallow as a promise the URL stays out of search.
The gain for site owners is fewer training rules once support is real. The token will not stop a fetch, change ordinary search visibility, or set a license fee. A reliable no before 2027 still depends on the older, company-specific lines.
