6 min read
You can now tell four of the biggest AI companies, in one short line of text, that you do not want your website used to train their models. It takes about 20 minutes, it costs nothing, and it will not hurt your Google ranking if you do it correctly. The catch is that it is a request, not a lock, and three of those four companies have not switched it on yet.
On October 30, Anthropic standards lead Kevin Kelley filed a draft called “Machine-Readable Training Preference Signal” with the Internet Engineering Task Force, the group that writes many of the rules the web runs on. OpenAI, Google, Microsoft and Anthropic have all said they will honor it. This guide walks you through the setup one step at a time, even if you have never opened a robots.txt file. A robots.txt file is a small public text file on your site that tells automated visitors what you would like them to do.
What the new token does, in plain English
The draft creates a label called __usage_ai-training. You place it in robots.txt, and it states a preference about one thing only: whether your pages may be used for generative AI training. Training is the stage where a company feeds content into a model so the model learns from it.
It does not stop anyone from visiting your site. The draft says plainly that the label is not a crawling rule. It also has nothing to say about live AI answers, such as when an assistant looks up your page to answer a customer. That is good news for a small business, because you can opt out of training and still be found and quoted by AI search. If that second part matters to you, read our beginner’s guide to getting mentioned in AI answers.
Timing varies by company, according to OpenAI’s announcement and press coverage of the other three:
- Anthropic: implementation started in October 2026.
- OpenAI: planned for the coming months.
- Microsoft: target date of February 1, 2027.
- Google: target of early 2027, when it will treat the label as a Google-Extended opt-out.
Adding the line now still makes sense. It is a two-minute edit, and it will already be in place as each company comes online.
Step 1: Decide what you actually want
Before you touch anything, sort your site into two piles.
- Pages you want spread widely: your services, pricing, location and press pages. If a model learns these, it may help people find you.
- Pages that are your product: paid guides, course lessons, original research, recipes or templates you sell.
If you sell your knowledge, opt out of the whole site and allow a few exceptions. If your site is mostly a brochure, the stakes are low. You could reasonably do nothing, or block only the folder that holds your paid material. There is no wrong answer here, and you can change your mind later.
Step 2: Find your robots.txt file
Type your domain into a browser and add /robots.txt to the end, like yourbusiness.com/robots.txt. You will see a short text file, or an error page if none exists yet. Copy what you see into a note so you have a backup.
Step 3: Open the editor for your platform
Where you edit the file depends on how your site was built:
- WordPress: WordPress creates this file automatically, so there is often nothing to open on the server. SEO plugins such as Yoast and Rank Math include a file editor, or you can upload your own file through your host.
- Shopify: edit the theme template named robots.txt.liquid.
- Wix: look for the robots.txt editor in the SEO settings.
- Squarespace and other closed builders: these may not allow direct editing. Check the current help page before you spend time hunting.
If Cloudflare sits in front of your site, check whether its managed robots.txt feature is on, because it can add to or change what visitors see.
Step 4: Add the lines
Add a blank line at the end of the file, then paste this as its own group:
User-Agent: __usage_ai-training
Disallow: /To keep certain pages open, add an exception. This version opts out everywhere except a press folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/Two mistakes to avoid. First, do not add Disallow: / under the line that reads User-Agent: *. That group talks to every crawler, including Google, and it could remove you from search. Second, do not assume that existing group already covers training. The draft says a wildcard group does not count, so a file that never names the label expresses no preference at all.
Step 5: Test it
Save, then open yourbusiness.com/robots.txt in a private browser window and confirm the new lines appear. If you use Google Search Console, its robots.txt report shows the version Google last fetched. Then spot check that your main pages still load and that nothing else in the file changed.
Step 6: Cover the gap with the rules that already work
Because the new label is still rolling out, you can add the older company-specific rules today. OpenAI’s crawler documentation separates GPTBot, which gathers training data, from OAI-SearchBot, which powers search. Anthropic’s training crawler is ClaudeBot, and Google uses a label called Google-Extended. To opt out of training without leaving AI search, block only those three:
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Google-Extended
Disallow: /OpenAI has said its existing crawler rules stay in place alongside the new signal, so these lines are not wasted effort.
What this cannot do for you
Stay honest with yourself about the limits. This is a preference, and it relies on companies choosing to respect it. It does nothing about content already inside trained models. Smaller scrapers that never agreed to anything can ignore it. Private licensing deals can override it. The draft itself is an individual submission with no formal standing at the IETF yet, and its details could change, so check back in the spring.
It is also not a copyright strategy. Your rights in your own work do not depend on this file, and they are a separate question from what you can claim over AI output, which we covered in our copyright myths piece. If your content is the core of your income, a short conversation with a lawyer is worth more than any robots.txt line.
Your 20-minute checklist
- Sort your pages into “spread widely” and “my product”.
- Back up your current robots.txt.
- Add the __usage_ai-training group, with any exceptions.
- Add GPTBot, ClaudeBot and Google-Extended for today’s coverage.
- Test in a private window, then set a calendar reminder for March 2027 to review it.
While you are in there, ask what else machines read on your site. Our plan for customers’ AI agents reading your policies is a sensible next stop.
