Robots.txt AI training opt-out: the new shared line

What did OpenAI, Google, Microsoft and Anthropic agree?

They agreed to read the same robots.txt line to learn whether a website wants its content used to train AI models. Until now, each company had its own crawler name or its own tag, so a site that wanted to say no had to write a rule for each one. With the shared line, you write it once.

robots.txt: The text file at the root of a website (yoursite.com/robots.txt) that tells crawlers which parts they may visit. Its format is set out in RFC 9309.
__usage_ai-training: A “policy token”: you write it where a crawler name would go, but it is not a crawler. It says which parts of the site you do not want used to train AI.
AI training: Using text, images or other content to change the learned parameters of a generative AI model. The IETF vocabulary defines it this way and leaves out models that do not generate content.

Why it matters: large language models learn from huge amounts of text, much of it taken from the web. For a publisher, a media outlet or a company with paid content, whether that content goes into training is a business decision. Until now there was no simple way to say it to everyone at once.

The four announcements came on Friday 30 October 2026: Google on its Search Central blog, Microsoft on the Bing blog, OpenAI on its website and Anthropic through Kevin Kelley, the author of the draft, on the IETF mailing list.

How do you write the line in robots.txt?

You write a normal robots.txt group whose User-Agent is the token. Disallow rules mark what you do not want used for training, and Allow rules mark what you do. This is the example from the draft itself:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Those three lines ask that nothing on the site be used for training, except the press kit folder. There are no new directives and no new syntax: it reuses the path rules every crawler already knows.

Three details are worth knowing before you copy it:

General rules do not count. The draft says a User-Agent: * group never applies to the token. If your robots.txt has no group for __usage_ai-training, it expresses no training preference at all, for or against.
It is not a crawling rule. The draft says so plainly: the token is not a directive about crawling. A search engine can still visit your site. What changes is what it may use the content for.
Private agreements win. If a site signs a licence with an AI company, that deal can override what robots.txt says.

Microsoft adds a per-page option for sites that want more control: a <meta name="Bingbot" content="ai-training:disallow"> tag or the matching HTTP header, which overrides robots.txt for that page.

When does each company start to honor it?

None fully supports it yet. Anthropic is the only one that says it has started, and it warns that it will take time to cover all its services. The other three gave dates from “the coming months” to March 2027.

Who will honor the shared line, and when

What each company announced on 30 October 2026

Company	When	Existing training control
Anthropic	Has started, step by step	Block ClaudeBot
OpenAI	In the coming months	Block GPTBot
Google	In 2027; read as a Google-Extended disallow	Block Google-Extended
Microsoft	From 1 February 2027	NoArchive and NoCache, which stop counting for this on 1 March 2027

Based on the announcements from Anthropic, OpenAI, Google and Microsoft. OpenAI timing as reported by Search Engine Journal and PPC Land.

Timeline of the shared robots.txt AI training opt-out: announcement by OpenAI, Google, Microsoft and Anthropic on 30 October 2026 and the first fixed date, Microsoft’s, on 1 February 2027, 94 days later — The shared line was announced on 30 October and Microsoft will read it from 1 February 2027. Infographic by Javier Sáez Hurtado, CC BY 4.0.

Google adds one rule. If your robots.txt opts out of training with the shared line and says nothing about Google-Extended, Google will treat it as if you had blocked Google-Extended. If you write an explicit Google-Extended rule, that rule wins. So you can say no to everyone except Google, or the other way round.

Microsoft changes the most. From 1 February 2027 it will read the shared line, and it splits two preferences: training, and grounding, which means using your page as a source in an AI answer. Grounding can only be set page by page, not in robots.txt. After 1 March 2027, Microsoft stops reading NoArchive and NoCache as training or grounding signals. Sites that use those tags on 15 December 2026 will get a temporary setting in Bing Webmaster Tools.

Will blocking AI training remove you from Google, ChatGPT or Gemini?

Not if you do it right. The shared line only covers training. All four companies keep crawling for training apart from crawling for search or for answering a person, each in its own way:

OpenAI. GPTBot is for training; OAI-SearchBot, for ChatGPT search; ChatGPT-User visits a page when a person asks the assistant to, for example through the ChatGPT Chrome extension. Its documentation says each setting is independent of the others.
Anthropic. ClaudeBot is for training; Claude-SearchBot, for search; Claude-User visits a page when someone asks for it.
Google. Google-Extended does not affect Google Search, but it does decide whether your site can appear with a link in Gemini app answers. Google says so in its post: sites that allow Google-Extended can appear and get clicks in those answers. If showing up in Gemini matters to you, this is the rule to think about twice.
Microsoft. Training and grounding are set separately, as described above, and neither changes how Bingbot crawls.

Table that separates training controls from search controls at OpenAI, Anthropic, Google and Microsoft, and shows that the shared __usage_ai-training line only covers training — What controls training and what controls search at each company. Infographic by Javier Sáez Hurtado, CC BY 4.0.

Google Search Central explains how robots.txt works. It is the video that goes with Google’s official guide.

The real risk is a typo. A Disallow: / under User-Agent: * instead of under the token tells every search engine to stay out of your site. It is a one-line change that can take weeks to undo.

What changes for companies in Europe?

In the European Union, the new line lands on ground that already had rules. The 2019 copyright directive allows text and data mining, which includes collecting content to train models, unless the rights holder has reserved that use “in an appropriate manner”, such as machine-readable means for content made public online (Article 4(3)).

The AI Act adds the other half. Since 2 August 2025, anyone who offers a general-purpose AI model in the EU must have a policy to comply with EU copyright law and, in particular, to identify and respect those reservations (Article 53). The code of practice the Commission published in July 2025 says crawlers should read and follow robots.txt as set out in RFC 9309.

What that rule lacked was a shared format. Each company read robots.txt with its own crawler names, and a site that wanted to reserve training could not be sure it had done it right for all of them. The shared line solves part of that, with two limits:

It is an individual draft. The IETF notes that it is not endorsed and has no formal standing. It may change, and the vocabulary it relies on (draft-ietf-aipref-vocab) does not yet reflect working group consensus.
It is not retroactive. The preferences that count are those in the robots.txt file in effect when the content was retrieved. What is already in a model stays there.

For a European company, the practical point is simple. If you want to reserve training, you now have a clearer way to put it in writing, and it fits what EU law asks of model providers. If you want AI tools to know and cite you, you don’t need to do anything. But it should be a decision, not an accident.

What would I check on my site today?

Before changing anything, I would run these checks, in this order:

What you have now. Open yoursite.com/robots.txt and see which crawlers you block. Many sites have blocked GPTBot since 2023 without remembering it, or carry rules from a plugin no one reviews.
What you want for each section. The public blog is not the same as the customer area, product pages or paid reports. The line accepts paths: you can say no to training in one folder and yes in another.
Whether you want to appear in AI answers. Blocking training does not remove you from search, but for Gemini, Google-Extended mixes the two. Decide which matters more to you.
Who else edits the file. Your CMS, your CDN or an SEO plugin can rewrite robots.txt. Check the file that is live, not the one you think is there.
A written record. A short internal note with what was decided, by whom and when. If an AI company asks tomorrow, or offers you a licence, you will know since when the reservation was in place.

The Google Search Central team answers what you can really put in a robots.txt file.

My view: a preference, not a lock

I think this is good news, mostly because it brings order. Four companies that compete on everything will read the same line, which makes life easier for anyone with a website. It is also a sensible way to give practical effect to an EU rule that already existed. That the author works at Anthropic, the company that has just filed to go public, and that the text sits at the IETF, open to everyone, helps too.

But a preference is not a lock. robots.txt works because serious companies choose to respect it. A crawler that does not want to follow it won’t. If some content must never reach third parties, the protection is not to publish it openly, not a line of text.

On my own site I chose the opposite of blocking: its robots.txt explicitly allows search crawlers and training crawlers. I write so that this information travels, and I want AI assistants to know it and cite it. That choice makes sense for a personal site that explains AI. I would not copy it as is for a company with paid content or customer data.

One last piece of advice, the same one I give about AI agents: a person should make this change by hand, and another person should check it. It is a file of a few lines that decides whether your site shows up in search engines. I would not leave it to an automatic plugin or to an AI agent without looking at the result. I explain the wider idea in my essay on how much autonomy to give an AI agent (in Spanish).

Disclosure: Convexify, the firm I founded, is part of the OpenAI Select Partner program. This article is based on public documentation from OpenAI, Google, Microsoft and Anthropic, on the IETF drafts and on the linked sources.

Key takeaways

The shared line is User-Agent: __usage_ai-training, with Disallow and Allow as usual. User-Agent: * rules do not apply to it.
Anthropic has started; OpenAI in the coming months; Google in 2027; Microsoft from 1 February 2027.
It covers training, not crawling: it does not remove you from Google or Bing. Google-Extended does affect linked answers in Gemini.
In the EU it gives shape to the machine-readable reservation of rights that the AI Act has required providers to respect since 2 August 2025.
It is neither retroactive nor a lock: content already used stays used, and a crawler that does not want to follow it won’t.

The article in one image

Visual summary in six ideas of the shared robots.txt AI training opt-out: how to write it, timeline, search, EU law and limits — You can reuse this infographic under a CC BY 4.0 licence: credit Javier Sáez Hurtado and link to this article.

Frequently asked questions

What is __usage_ai-training?

It is a policy token that you write in robots.txt as if it were the name of a crawler (User-Agent: __usage_ai-training). With the usual Disallow and Allow rules, the site says which parts it does not want used to train AI models and which parts it does. It is defined in an IETF draft dated 30 October 2026.

How do I stop AI companies from training on my website?

Today you need two things: block each company’s training crawler in robots.txt (GPTBot, ClaudeBot, Google-Extended) and add the shared group User-Agent: __usage_ai-training with Disallow: /, which the four companies will honor between now and 2027. Neither removes content that was already used for training.

Will blocking AI training remove my site from Google or ChatGPT search?

No. The shared line only covers training, and Google and Microsoft keep training separate from search. At OpenAI, GPTBot (training) and OAI-SearchBot (search) are set separately. One catch: Google-Extended also decides whether your site can appear with a link in Gemini app answers.

When will OpenAI, Google and Microsoft start to honor it?

Anthropic says it has already started. OpenAI says in the coming months, Google says 2027 and Microsoft will read it from 1 February 2027. Microsoft gives sites until 1 March 2027 to move from NoArchive and NoCache to the new preferences.

Do AI companies have to respect robots.txt in Europe?

Since 2 August 2025, providers of general-purpose AI models in the EU must have a policy to respect rights reservations made in a machine-readable way (Article 53 of the AI Act and Article 4 of the copyright directive). The Commission’s code of practice names robots.txt. The new line is not law: it is a shared way to express that reservation.

Does it remove my content from models that are already trained?

No. The draft says the preferences that count are those in the robots.txt file in effect when the content was retrieved. Adding the line now does not take earlier content out of existing models.

Sources

Javier Sáez Hurtado is the founder of Convexify, an Andorra-based boutique AI firm, and Head of AI Heroes at Learning Heroes. Podcast appearances · LinkedIn · YouTube channel

One robots.txt line to opt out of AI training: what OpenAI, Google, Microsoft and Anthropic agreed