Publishers have long had to tell every AI company's crawler, one by one, not to use their work for training. On October 30 and 31, 2026, four of the biggest AI companies said that job is about to get simpler. OpenAI, Google, Microsoft and Anthropic pledged to honour a single, shared signal that lets a website say, in one line, that its content should not be used to train AI models. It is not a law, and it will not work everywhere on day one, but it is the first time the biggest players have agreed on the same opt-out language.
What the four companies actually pledged
Instead of separate blocking instructions for each company's bots, a publisher will be able to write one standardised instruction that any adopting AI company recognises. According to Channel NewsAsia and SiliconANGLE, one entry in a site's robots.txt file can then speak to all four crawlers at once. Meta also expressed support through spokesperson Andy Stone, though the core pledge centres on OpenAI, Google, Microsoft and Anthropic.
The one line that does the work
The shared token is usage_ai-training. It lives in robots.txt, the standard file site operators already use to publish rules for automated crawlers. The token is defined in an individual draft at the Internet Engineering Task Force, titled Machine-Readable Training Preference Signal, authored by Kevin Kelley of Anthropic and dated October 30, 2026. That status matters: an individual draft is an early stage of standardisation, not yet a formal standard. Publishers are being asked to trust a shared convention while the formal standard catches up.
When each company will honour it
The timelines are not the same, and that uneven start is the first thing publishers should note. Anthropic has already started rolling out support for the token. OpenAI said it will roll its support out in the coming months. Microsoft will implement it beginning February 1, 2027, and Google sometime in 2027. A publisher adding the token now should therefore expect uneven enforcement through much of 2027, and check each rollout date as it passes.
What it does NOT do
This is where careful reading matters. The new setting applies only to future training of models. It does not prevent bots from crawling a site otherwise, and content scraping will be allowed by default until a publisher changes the setting, according to Microsoft. It also does not replace the controls companies already provide, and OpenAI says the new signal does not guarantee exclusion from training. It covers training use specifically, while search indexing or retrieval follow separate rules unless a site restricts those as well. It is a preference signal relying on voluntary compliance, not an enforcement mechanism. That distinction is similar to provenance tools, such as the watermark detector Google just opened to everyone, which helps identify AI-made content but does not by itself stop training use.
Why publishers pushed for this
The announcement comes after years of tension between publishers and AI companies over training data. Some publishers, like the New York Times, have taken the matter to court, and the Times sued OpenAI and Microsoft in December 2023, alleging millions of its articles were used without authorisation or payment to train ChatGPT, a claim the companies have contested by invoking fair use. That fight is why a single opt-out matters: today each publisher must negotiate, block, or litigate company by company. At the same time, Microsoft and Google warned opting out could reduce a publisher's visibility in AI apps, so there is a real trade-off between control and visibility.
What to watch next
Three questions will decide whether this pledge becomes practical protection or just a press release. First, will other AI companies adopt the same token, or will publishers still need separate rules for everyone else? Second, will the IETF draft move from an individual draft toward a formal standard that companies can be held to more clearly? Third, how will regulators treat a voluntary signal alongside new laws? Regulation is already moving in parallel, as shown by New York's new limits on companion AI chatbots for under-18s, and a voluntary opt-out may end up sitting next to, not instead of, legal rules. Meanwhile the same companies are still racing on models, with Google unveiling Gemini 4 Carbon as its strongest AI yet, showing why training-data rules are now a front-line issue.
One line in robots.txt will not settle the training-data debate, but it does change the starting point: publishers no longer have to repeat themselves four times to be heard by the four biggest AI companies. The test now is rollout, adoption beyond the four, and whether the signal is honoured in practice. Would you opt your site out of AI training if it meant less visibility in AI answers, or is visibility worth the trade? Tell us in the comments.
Comments
Post a Comment