A single signal instead of per-crawler rules
OpenAI, Google, and Microsoft announced on October 30, local time, their intention to support a common mechanism that allows website operators to communicate to multiple artificial intelligence companies at once whether their content may be used to train models. ITmedia AI+ reports this, according to which Anthropic is also proceeding with implementation. The mechanism originates from an Internet-Draft presented the same day to the IETF by Kevin Kelly, Anthropic's head of standardization, and is based on the work of the same organization's AIPREF group.
Until now, website operators had to identify each company's crawler and write separate rules for each. The new approach calls for inserting a shared policy token, "__usage_ai-training", in robots.txt under the "User-Agent" line: "Disallow" directives indicate refusal of use for training, while "Allow" expresses consent. The example cited by the source shows how the entire site can be excluded except for a specific folder.
What really changes for websites and what remains excluded
The source clarifies a decisive point: this signal does not block crawlers from scanning content. Traditional per-crawler controls remain valid for limiting access. The token concerns only the use of content to modify the already trained parameters of a generative model; according to the definition cited, it does not cover training aimed at uses other than generation. In the absence of the token, it is assumed that the site has expressed no preference.
The choice not to introduce new directives or new syntax should avoid effects on existing parsers. The possibility is also recognized for companies to give precedence to their own control tools over the common signal or to apply different treatment based on specific contracts. The draft, the source emphasizes, was filed in a personal capacity and currently has no formal status in the IETF standardization process.
Differing timelines and the standardization question
The announced timelines do not align. OpenAI aims to implement support within a few months and has said it will publish details at the time of implementation, keeping existing robots.txt controls in place and inviting other AI companies to respect the shared signal. Google is targeting 2027 and specified that if a site refuses training through the shared signal without explicitly indicating Google-Extended, that refusal will also be considered valid for that tool. Microsoft expects to read and apply the new settings from February 1, 2027.
The source also reports that Google links the initiative to requests from publishers, civil society groups, and regulatory authorities and considers it consistent with the code of conduct for general-purpose AI model providers envisaged by the EU AI Act. Microsoft indicated that, in addition to the signal in robots.txt, it will introduce page-level controls via meta tags and HTTP headers, with precedence over robots.txt, and that the new settings will replace NoArchive and NoCache, with migration requested by March 1, 2027. The actual adoption by other companies and any evolution of the draft at the IETF remain to be verified.
