Major AI Firms Back Unified Standard for Publishers to Opt Out of Data Scraping

In a significant development for digital content rights, OpenAI, Google, Microsoft, and Anthropic have announced their commitment to adopting a new technical standard that allows publishers to opt out of AI data scraping. This unified control mechanism enables website owners to apply their AI training preferences across multiple AI companies simultaneously, eliminating the need to configure settings individually for each provider. The move addresses the growing friction between content creators and the automated crawlers that feed data into large language models. The proposed solution, known as the shared “policy token” signal, was introduced by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF). This token will be embedded within a website’s robots.txt file, a file that has traditionally governed which web crawlers are permitted to access a site for search indexing, archiving, or AI training. Currently, publishers must signal their preferences separately to each AI entity, a process that is both cumbersome and inefficient. By centralizing this control, the new standard streamlines the process for publishers seeking to protect their intellectual property. This announcement arrives at a critical juncture as publishers continue to grapple with the influx of AI crawlers scraping their sites. Previous attempts to combat unauthorized scraping have included the Really Simple Licensing (RSL) standard, which allows publishers to charge for crawler access, and Cloudflare’s default blocking of AI bots. The new standard offers a more cooperative and standardized approach to these challenges, aiming to establish a clear protocol for data usage across the industry. Microsoft has indicated it will offer granular controls, allowing publishers to opt out of AI training or AI grounding—the process of pointing a model to verified sources to reduce hallucinations—while ensuring their sites remain visible in Bing search results. Google, which has previously implemented similar controls, plans to support the new standard next year, while OpenAI stated it will implement the change in the coming months. Anthropic has already adopted the signal, signaling a rapid industry shift toward respecting publisher sovereignty over their data. Meta’s communications head, Andy Stone, also expressed support for the new setting, describing the creation of a common signal as an important step. However, Meta did not explicitly confirm whether it would honor the token. Stone stated, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” adding that the company looks forward to collaborating with other AI firms on standards implementation. This broad coalition of tech giants suggests a potential end to the fragmented landscape of AI data governance.