One robots.txt line will opt your site out of AI training at Google and OpenAI

Anyone who wanted to keep a website out of AI training has had to write a separate rule for every company, learning names like GPTBot, ClaudeBot and Google-Extended one by one. Google, OpenAI, Anthropic and Microsoft now say they will all read the same line instead: a shared “policy token” in robots.txt, the small text file that has told search engines where not to go since the early web. Written once, that entry will carry the same answer to four of the largest AI developers.

The idea comes from Kevin Kelley, a technical lead for standards at Anthropic, who submitted it to the Internet Engineering Task Force, the body that writes the internet’s technical rules. Its neat trick is that it invents no new grammar. A site owner writes an ordinary rule group, but the User-Agent line names a use rather than a crawler: __usage_ai-training. Under it, “Disallow: /” means do not train on anything here, and an “Allow” line can carve out exceptions, such as a press kit a company would be glad to see a model learn from.

For a blog, a shop or a newsroom, the most useful detail is what the line leaves alone. It states a preference about how content is used, not about who may visit, so search keeps working. Google says that when a site opts out with the shared token and does not mention Google-Extended separately, it will read that as a Google-Extended block, which covers Gemini training and leaves the page in Google Search. Microsoft makes the same promise for Bing: the setting does not change whether a page can appear in its results.

Microsoft goes a step further than the others. Next to training, it is adding a separate grounding preference, set page by page through a meta tag or an HTTP header, which decides whether a page can feed the answers that Bing, Copilot and Microsoft 365 generate on the spot. A site could refuse training and still allow grounding, keeping its citations and links inside AI answers. Microsoft warns that saying no to grounding will “significantly reduce” a site’s visibility in those experiences, and that neither choice guarantees inclusion, traffic or payment.

There is a catch hiding in the file most sites already have. Kelley’s draft deliberately makes the familiar wildcard rule, “User-agent: *”, blind to the new token, so a robots.txt that shuts out every bot today expresses no AI-training preference at all. Owners who want out have to add the new group by name. The signal also only looks forward: whatever the companies crawled before the line appears is already in their training sets, and nothing in the draft forces any company to obey it. It is a request that four firms have promised to respect, not a lock.

The most conspicuous absence is Meta. Its head of communications, Andy Stone, told The Verge that a common signal reaching several providers at once is “an important step”, but did not say whether Meta will honour it. The timing is awkward. On the same day the four companies announced their support, Meta’s updated business terms took effect, giving it permission to send crawlers to websites tied to Facebook, Instagram and WhatsApp business accounts and to use what they collect to train its products.

The rollout is uneven. Anthropic has already started implementing the token, though Kelley wrote that covering all of its services will take time. Microsoft will honour it as it crawls from February 1, 2027, and after March 1 will stop reading its older NoArchive and NoCache tags as training or grounding signals. Google aims to support the token in 2027 and promises progress updates, while OpenAI has committed only to the coming months. The proposal still has to pass through the IETF’s AI Preferences working group before it becomes a formal standard, and Google notes that it fits the EU’s code of practice for general-purpose AI models, which already treats robots.txt as the main way publishers state machine-readable preferences.

What makes the moment remarkable is the size of the fix. After years of lawsuits and licensing fights over who owns the open web, the first shared answer is a few characters in a plain text file that anyone can edit in a minute. How much it protects will depend on who signs up next, and the company with the most business pages on its platforms has not yet said yes.

Tags: OpenAI, Google, Anthropic, meta, microsoft, AI training