OpenAI, Google, Microsoft, Anthropic back one AI training opt-out
Four AI majors will honour a single robots.txt token that lets publishers refuse model training. Here is what it means for Indian news sites and labs.
Key points
- OpenAI, Google, Microsoft and Anthropic have committed to honouring a shared robots.txt token that lets sites opt out of AI training.
- Anthropic has already adopted the token, Google and Microsoft plan support next year, and OpenAI says it will follow in the coming months.
- Indian publishers gain a simpler way to refuse training while keeping search traffic, though content already scraped is not covered.
Photo: Florian Hirzinger - www.fh-ap.com / Wikimedia Commons, CC BY-SA 3.0 (cropped). Representational image.
The News
Four of the largest AI developers - OpenAI, Google, Microsoft and Anthropic - have committed to honouring a single signal that lets website owners refuse AI training across all of them at once. The signal, a shared "policy token", sits in a site's robots.txt file, the long-standing text file that tells crawlers what they may and may not collect.
The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, and is being taken forward through the Internet Engineering Task Force (IETF). Until now, a publisher wanting to keep its pages out of model training had to list each AI company's crawler separately. Under the new approach, one entry would cover every participating firm, and each of the four says its crawlers will stay away from sites that carry it.
Anthropic has already adopted the token. Google and Microsoft intend to support it next year, while OpenAI has said only that it will do so in the "coming months". Microsoft also plans finer controls that separate AI training from AI grounding, and says Bing search visibility will not be affected by either choice. Meta's communications head, Andy Stone, welcomed the idea but did not say whether Meta would comply.
Why It Matters
The robots.txt convention dates back to 1994 and was built for search engines, which sent readers back to the sites they indexed. AI systems break that bargain: they absorb the content and often answer the reader directly. Publishers have responded with blunt tools. Cloudflare now blocks AI crawlers by default, and the Really Simple Licensing (RSL) standard lets sites charge crawlers for access.
A common token is the first time the major model builders have agreed on one opt-out mechanism rather than each publishing its own crawler names. The closest precedent is the 2005 "nofollow" attribute, adopted jointly by the big search engines against comment spam. If the IETF process yields a formal standard, refusing AI training could become routine for webmasters.
The limits are equally clear. The token is voluntary, it applies only to future crawling, and the holdouts - including Meta for now - can ignore it. Content already scraped stays inside existing models.
Indian Angle
For Indian news publishers the timing is pointed. News agency ANI's copyright suit against OpenAI, filed in the Delhi High Court in November 2024, turned on exactly this question: whether scraping publicly available news for training needs permission. A shared opt-out helps going forward, but Indian media houses are likely to see it as a partial fix rather than an answer to claims over past use.
There is also a cost question for smaller Indian sites. Regional-language publishers and independent finance portals rarely have engineering teams tracking dozens of crawler names. One line in robots.txt is a far lower bar, and it lets them keep Google Search traffic while declining training, a split many Indian outlets depend on.
Indian model builders such as Sarvam and Krutrim, which rely heavily on Indian-language web text, will face pressure to state whether they will honour the same token. With MeitY still shaping AI governance guidelines, voluntary compliance by domestic labs may become a reputational test.
FAQ
What exactly is the policy token?
It is a shared entry in a website's robots.txt file signalling that the owner does not want its content used for AI training. One instruction replaces separate blocks for each AI company's crawler.
When will the four companies start honouring it?
Anthropic has already adopted it. Google and Microsoft have said they plan to support it next year. OpenAI has given a looser commitment, saying it will implement the signal in the coming months. Until then, existing crawler-specific blocks remain the safer option.
Will blocking AI training hurt my search ranking?
According to Microsoft, sites will continue to appear in Bing results whatever their AI preference. Google already offers controls that separate AI training and AI search features from ordinary search visibility.
Does it remove content that has already been scraped?
No. The token only governs future crawling by companies that choose to honour it. Material already collected and used to train existing models is not affected, which is why ongoing disputes such as the ANI case in the Delhi High Court remain relevant.
This story was reported by The Verge. Read the full original coverage at The Verge.