Presented by Omnia
Search is changing. If you’re part of a small team that’s expected to do more with less, Omnia is for you.
Omnia helps you track your brand’s visibility across the leading AI search platforms: ChatGPT, Perplexity, Google AI Overviews, and Google AI Mode. See how you compare with competitors, which sources these platforms cite, and where you have opportunities to improve your presence.
Omnia also helps you make concrete improvements, with an autonomous agent and built-in project management. Turn data into action, prioritize optimizations, and keep your AEO strategy on track without overwhelming your team.
Try Omnia free for 14 days and get 20% off your first year when you decide to upgrade.
On October 30, 2026, Google, OpenAI, and Microsoft announced that they would honor a shared robots.txt line, __usage_ai-training, so any site can ask that its content not be used to train AI models. Robots.txt is the text file every site publishes to tell crawlers (the programs that roam the web and download pages automatically) what they’re allowed to visit. For anyone hoping to block AI crawlers, the news comes with two caveats. The line doesn’t work yet: Microsoft will honor it starting February 1, 2027, Google sometime in 2027, and OpenAI “in the coming months.” And it doesn’t stop crawling: the technical draft that defines it clarifies that it “is not a directive about crawling behavior.”
The line settles one technical decision, about training, and leaves the strategic one open: what a brand actually wants from AI platforms. Aaron Marco Arias, of Postdigitalist, framed it in a post on X with two paths. The first is AEO (answer engine optimization) or GEO (generative engine optimization), two names for the practice of optimizing content so ChatGPT, Gemini, and other assistants cite it as a source in their answers. The second is blocking crawlers and negotiating a licensing deal with AI companies, an agreement to get paid for access to the content. According to Arias, which one works depends on where the brand sits relative to the platform. A small brand needs the AI to send it demand. A powerful brand, on the other hand, as Arias put it, “knows that when its audience asks AI about X, it will expect to see itself in the answer,” and that gives it something to negotiate with.
How to block AI crawlers in robots.txt
These lines ask that your site not be used for training while leaving ChatGPT search open. They go in robots.txt, at the domain root:
# block AI crawlers (training): shared signal. Google, Microsoft: 2027; OpenAI: "coming months"
User-Agent: __usage_ai-training
Disallow: /
# Training + grounding in Gemini Apps (Google)
User-Agent: Google-Extended
Disallow: /
# OpenAI training (does not affect ChatGPT search)
User-Agent: GPTBot
Disallow: /Five rules for using them:
- The shared line only covers training. It doesn’t stop crawling, and a blanket rule with User-Agent: * doesn’t count: the draft requires naming the token explicitly. A file without the token expresses no preference at all.
- To keep showing up in ChatGPT search, don’t block OAI-SearchBot, the bot OpenAI uses for its search product.
- Google ties training to grounding, which is when an assistant checks pages in real time to support its answer. Blocking Google-Extended also removes you from grounding in Gemini Apps, but not from Google Search.
- Microsoft separates the two. Grounding is configured page by page, with Disallow or DisallowExtended. If you currently use NoArchive or NoCache, switch them before March 1, 2027.
- Your existing rules still apply, and a private licensing agreement can override whatever robots.txt says.
Robots.txt is a voluntary request, so actual blocking happens at the CDN level (the network that delivers the site and filters traffic before it reaches the server) or in the firewall, with tools like Cloudflare’s AI Crawl Control, which lets you block crawlers one by one and see which ones ignore robots.txt.
What changes with Google, OpenAI, and Microsoft’s shared line
Until now, every AI company had its own bots, and every site had to write a separate rule for each one. Google describes the change this way: “Rather than juggling separate instructions for each AI developer, site owners will be able to use a common policy token within robots.txt to set general preferences toward participating AI providers.”
The token comes from an individual draft submitted to the IETF (Internet Engineering Task Force, the body that publishes internet technical standards) by Kevin Kelley, an Anthropic employee, the company that builds the Claude assistant. Search Engine Journal, an SEO trade publication, noted that the document doesn’t represent an official IETF standardization effort. The draft expires on May 3, 2027. Anthropic didn’t publish its own announcement; according to Search Engine Journal, Kelley said the company has begun building support for it.
The draft defines AI training as “using a resource to modify the learned parameters of a generative AI model.” Models that don’t generate content, such as classification, ranking, or scoring models, are excluded.
Each company connects it to its own controls in its own way. If a site uses the shared line without mentioning Google-Extended, Google’s control for Gemini training, Google will treat it as if the site had blocked that control too. OpenAI says its existing controls “will continue to apply.” Microsoft adds a per-page AI Grounding preference, via DisallowExtended, that applies only to its own products, and warns that if a site expresses no grounding preference, its content “may be used.”
Path 1: getting AI to cite you (AEO/GEO)
To show up as a source in an AI answer, search and grounding bots need to be able to get in. At OpenAI, the bots are independent: GPTBot handles training and OAI-SearchBot feeds ChatGPT search. According to OpenAI’s documentation, sites that block OAI-SearchBot “will not appear in ChatGPT search results.” A site can shut down training and still be citable.
At Google, the math is different. According to its crawler documentation, Google-Extended governs training for future Gemini models and also grounding in Gemini Apps and Vertex AI, Google Cloud’s AI platform, so opting out of one means opting out of the other: “Sites that specifically allow Google-Extended may appear in responses and receive clicks through grounding in Gemini Apps.” What Google-Extended doesn’t touch is Google Search: it doesn’t affect a site’s inclusion or act as a ranking signal.
Microsoft, by contrast, lets you opt out of training while staying in grounding, page by page.
For a small brand, this is the natural path, and its technical foundation is SEO: a site that machines can read, as we explained in our guide to agentic search and how AI agents choose brands, and a presence across every surface where people search, the idea behind Search Everywhere Optimization. The path has its gray zone too: writing instructions for AI that human readers never see, like the ones that turned up in WhatsApp’s help pages.
Path 2: block AI crawlers and negotiate
The second path demands something most brands don’t have: a platform that needs them. Arias points to the Condé Nast publishing group and Mercado Libre, Latin America’s largest e-commerce platform, as examples of brands with that kind of leverage.
The Mercado Libre case is what set off the discussion. On October 9, X user @nicoproducto asked: “Why doesn’t Mercado Libre let LLMs access the product information posted there?” (LLMs are the language models behind assistants like ChatGPT or Claude). mercadolibre.com.ar’s robots.txt, checked on November 2, has Disallow: / (the instruction not to visit any page) for Amazonbot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-User, GPTBot, and ChatGPT-User. It has no rules for Google-Extended or OAI-SearchBot. We don’t know why, and we found no information suggesting Mercado Libre is negotiating with OpenAI.
One deal that already exists in Spanish-language media is Prisa Media’s, the Spanish group that owns El País. On March 13, 2024, OpenAI announced a deal with Prisa to show content from El País, Cinco Días, AS, and El HuffPost in ChatGPT, and to also use it for model training. elpais.com’s robots.txt, checked on November 2, blocks ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, CCBot, and Amazonbot, among others, and has no rules for GPTBot or Google-Extended. A comment in the file reads: “OAI-AdsBot remains allowed.” The evidence isn’t enough to claim one thing explains the other. The IETF draft does anticipate that scenario: “Private agreements MAY override preferences expressed through this mechanism.”
Two other platforms made moves the same week. On October 29, Reddit, the forum and community site, announced it’s shifting to a mostly logged-in experience while continuing to license its data, according to Axios and TechCrunch. Its CEO, Steve Huffman, told Reuters, in a story published by Investing.com: “The reality is that AI Mode doesn’t drive traffic, not for us, not for anyone.” AI Mode is Google’s AI-powered search mode.
A day later, the CMA (Competition and Markets Authority, the UK’s competition regulator) opened an investigation into Meta’s commercial terms, which the regulator said could allow Meta to use content from other companies “without those companies being able to choose whether or how their content is used.” Meta had paused those terms on October 28, and the CMA clarified that it hadn’t made any decision yet.
Europe and Latin America
Google presents the shared line as consistent with the European Union’s Code of Practice for providers of general-purpose AI models. That’s Google’s reading of it. In its copyright chapter, the code states that signatories commit to using crawlers that read and follow robots.txt instructions per RFC 9309, the technical specification for the protocol.
In Argentina, the robots.txt files for Clarín, La Nación, and Infobae had no rules for AI bots when we checked on November 2. That doesn’t mean they aren’t blocking them: they might be doing so at the CDN or firewall level, and we did not verify that.
What to watch in 2027
- February 1: Microsoft starts honoring the new training and grounding preferences.
- March 1: NoArchive and NoCache stop counting as preferences at Microsoft.
- Early 2027: Google turns on support, according to its crawler documentation. OpenAI hasn’t given a date.
- May 3: the IETF draft expires.
The shared line gives any site a cheap way to say no to training. What it doesn’t give is negotiating power. For brands that don’t have an audience expecting to see them in the answer, the useful question is the one Arias closed his post with: “don’t ask what OpenAI could do for you, ask what you could do for OpenAI.” OpenAI already has its own answer for a lot of them: as of October 21, it sells ads in ChatGPT across 16 Latin American markets.
Frequently asked questions
What is the __usage_ai-training line in robots.txt?
The __usage_ai-training line is a shared robots.txt signal that asks that a site’s content not be used to train generative AI models. It comes from an individual draft submitted to the IETF by Kevin Kelley, of Anthropic. Google, OpenAI, and Microsoft announced on October 30, 2026, that they would honor it: Microsoft starting February 1, 2027, Google sometime in 2027, and OpenAI in the coming months.
Does the __usage_ai-training line block AI crawlers?
The __usage_ai-training line expresses a preference about training and still lets crawlers into the site, because the draft that defines it clarifies that it is not a crawling directive. To block AI crawlers via robots.txt, you need to write separate rules for each bot, such as GPTBot or ClaudeBot. To actually enforce a block, you need to filter them at the CDN or firewall level, since robots.txt is voluntary.
Does blocking Google-Extended remove a site from Google Search?
Blocking Google-Extended doesn’t remove a site from Google Search, according to Google’s crawler documentation, and it doesn’t act as a ranking signal either. It does remove the site from training for future Gemini models and from grounding in Gemini Apps and Vertex AI, which is when Gemini checks pages to support its answers. Google ties the two together: a site can’t opt out of training while staying in that grounding.
Can a site appear in ChatGPT search without OpenAI training on it?
A site can appear in ChatGPT search without OpenAI training on its content, because according to OpenAI’s documentation its bots work independently. GPTBot is the training bot and can be blocked in robots.txt. OAI-SearchBot feeds ChatGPT search, and OpenAI warns that sites that block it won’t appear in that search product’s results.
How do you block AI crawlers that don’t respect robots.txt?
AI crawlers that don’t respect robots.txt have to be blocked at the CDN or firewall level, since robots.txt is a voluntary request the bot can ignore. Cloudflare, for instance, offers AI Crawl Control on all its plans: it lets you set blocking rules for each crawler and see which ones violate the site’s robots.txt instructions.