Google joins OpenAI and Anthropic in promising new scraping controls

Google, Anthropic, OpenAI, and Microsoft have committed to honoring a new robots.txt directive that allows publishers to control scraping for generative AI training. Rather than requiring separate rules for each crawler, the companies are adopting a common standard under consideration at the Internet Engineering Task Force. As detailed in engadget.com's report, the shared convention establishes a single common signal for AI developers.

This report details how the directive works, the exact syntax operators must implement, and the rollout schedules across participating firms. It is written for website owners, engineers, and digital publishers who want to block automated training crawls without harming regular search indexation. The unified standard isolates training data collection from general indexing, providing a predictable framework for domain management.

Google joins OpenAI and Anthropic: what actually changed

The shift eliminates the need for website operators to configure distinct robots.txt entries for every individual AI crawler. Previously, managing permissions meant tracking each company's bespoke bots and writing independent exclusion rules. Under a proposal submitted to the Internet Engineering Task Force by Anthropic technical lead for standards Kevin Kelley, Google, Anthropic, OpenAI, and Microsoft are adopting a unified standard. The companies will recognize a single common signal to govern automated data harvesting for generative model training across public web services.

By unifying behind an IETF standard, the participating firms establish a common protocol that replaces proprietary opt-out directives. Website owners no longer need to update their configurations whenever a provider introduces a new crawler or alters existing scraping tools. Instead, site operators can declare their preferences once using a single entry. The four companies have committed to honoring this directive across their future ingestion pipelines, significantly simplifying web administration and establishing consistent scraping boundaries across the broader artificial intelligence commercial ecosystem.

Google joins OpenAI and Anthropic: how it works

The implementation operates through a new user-agent section in the standard robots.txt file. Publishers specify the directive as User-Agent: usage_ai-training, followed by standard permissions. Adding Disallow: / blocks participating crawlers from gathering data across the entire website for model training. Operators can also define granular exceptions by including an Allow path, such as Allow: /press-kit/, permitting models to ingest specific public materials while restricting the rest of the site. This syntax fits directly into existing server infrastructure without requiring custom scripts or specialized networking configurations.

The new directive does not block crawling entirely or prevent pages from appearing in Google search results. Google will interpret the instruction specifically as a stop sign for its Google-Extended crawler, which collects materials used to train Gemini models. Standard Googlebot crawlers that index content for web search will continue accessing pages normally. This separation enables publishers to opt out of generative AI training without sacrificing organic traffic or search visibility, resolving a persistent long-standing operational tension between general web indexation and automated AI data harvesting.

Google joins OpenAI and Anthropic: why it matters now

Standardization resolves an administrative challenge created by the rapid rise of generative artificial intelligence systems. Without an industry standard, website operators had to research individual crawlers and continually update robots.txt rules whenever companies deployed new automated scrapers. Kevin Kelley's proposal at the IETF gives the industry a formal, open-standards baseline. Having four major AI developers support a single signal creates clear expectations across the web and simplifies legal and technical compliance for publishers who want to protect original published editorial content.

However, the directive has a critical operational limitation: it cannot undo past scraping. If Google, OpenAI, Anthropic, or Microsoft previously crawled a website, they already possess that data, and those historical archives remain in existing model datasets. Implementing the directive only stops future scraping of websites or specific pages once the rules take effect. Site operators cannot use robots.txt to retroactively remove their content from models that have already completed training, meaning the protection applies exclusively to upcoming automated data crawl cycles.

Google joins OpenAI and Anthropic: who is affected

Website owners, webmasters, and online publishers are directly impacted by this change. Instead of maintaining sprawling robots.txt files filled with separate Disallow directives for individual crawlers, administrators can apply the usage_ai-training user agent once. This reduces configuration errors and administrative overhead. Publishers who rely on search discovery can protect their proprietary articles while still permitting search bots to index pages. Organizations can also use Allow paths to permit training access for specific sections, such as press kits, giving them granular control over their public web assets.

The artificial intelligence companies themselves are also affected, as they must adjust their data-gathering infrastructure to respect the directive. Google must configure its Google-Extended crawler to recognize the new standard and respect disallow rules when collecting training material for Gemini. Anthropic, Microsoft, and OpenAI must similarly update their automated scrapers to obey the universal instruction. This requires each participating company to align its data collection practices with the IETF proposal, ensuring that training systems do not bypass publisher preferences during future web scraping runs.

Google joins OpenAI and Anthropic: what to watch

The timeline for adoption varies significantly across the four participating companies. Anthropic has already started supporting the new directive, meaning its crawlers should already respect the usage_ai-training signal. Microsoft announced it will begin supporting the directive on February 1, 2027. Google stated it plans to support the directive in early 2027, integrating the signal with its Google-Extended crawler. OpenAI has pledged to implement the directive in the coming months, though it has not yet provided a specific calendar date for its rollout.

Because implementation dates are staggered throughout late 2026 and early 2027, website administrators must continue maintaining individual crawler rules during the transition. Deploying the usage_ai-training directive immediately prepares a website for compliance, but older bot rules remain necessary until Google, Microsoft, and OpenAI complete their rollouts. Industry observers should also monitor the IETF review process for Kevin Kelley's proposal to see whether other AI developers adopt the standard, which would further solidify robots.txt as the primary governance mechanism for AI data collection.