OpenAI, Google and Microsoft will honor one robots.txt rule that opts sites out of AI training
A shared token, __usage_ai-training, lets a site say no to AI training by every participating company at once. It doesn't cover AI answers.
The skim
- OpenAI, Google and Microsoft said Friday they will honor one shared robots.txt rule that lets a website opt out of AI training by every participating company at once. Search Engine Land reports that Anthropic is also on board, and an Anthropic engineer wrote the draft spec.
- The rule is a new policy token,
__usage_ai-training. It covers training only. It doesn't block crawling, search, or AI answers that quote your pages. - Nothing changes yet. Microsoft starts honoring it on Feb. 1, 2027, Google says it aims to support it in 2027, and OpenAI says "in the coming months."
What was announced
Right now, a publisher who doesn't want its work used to train AI models has to block each company's crawler by name: GPTBot, Google-Extended, ClaudeBot and a growing list of others. The new signal collapses that into one entry. It comes from the Internet Engineering Task Force's AI Preferences working group and is written up in an Internet-Draft by Kevin Kelley of Anthropic. A site that wants out adds two lines to its robots.txt: User-Agent: __usage_ai-training and Disallow: /. It can also opt out only some folders.
"Instead of managing separate instructions for every AI developer, website owners will be able to use a shared 'policy token' within robots.txt to set broad preferences across participating AI providers," Google's John Mueller wrote on the Search Central blog. If a site uses the token to opt out and doesn't mention Google-Extended, Google will treat that as blocking Google-Extended. OpenAI said its existing crawler controls will keep working alongside the new signal.
Microsoft went furthest. It's adding a page-level training setting and a separate "AI Grounding" setting that decides whether a page can feed AI answers in Bing, Copilot and Microsoft 365. A third option, DisallowExtended, allows grounding in Microsoft's own products but not in the grounding services it sells to other companies. These settings replace Bing's old NoArchive and NoCache tags, and sites that rely on those tags have until March 1, 2027, to switch.
Why it matters
For publishers, this fixes a real chore. They can now make one choice for every participating company instead of keeping up with new crawler names. But read the fine print. The signal only covers training. Opting out won't keep a page out of AI-written search answers, which is the use that's costing sites like Reddit traffic. Google notes that sites that specifically allow Google-Extended are eligible to appear, and get clicks, in Gemini app answers. Microsoft is blunt about the trade-off: "Neither choice guarantees inclusion, traffic, or payment."
The signal is also voluntary. The draft "is not endorsed by the IETF and has no formal standing," according to its own datatracker page, and it only binds the companies that choose to follow it.
What's next
The working group will keep refining the draft, which expires in May 2027 unless it's updated. Site owners don't need to do anything yet, but the two-line opt-out is already published, and adding it now costs nothing. Anyone using Bing's NoArchive or NoCache tags should note the March 1 deadline.
Sources: OpenAI, Google Search Central, Microsoft Bing, IETF datatracker and Search Engine Land.
