Google, Anthropic and others to support new AI crawling opt-out standard

Google LLC, Microsoft Corp., Anthropic PBC, and OpenAI Group PBC announced plans to support an upcoming web crawling standard. As reported in siliconangle.com's report, the framework enables website operators to prevent artificial intelligence developers from using their content for model training. Rather than requiring distinct entries for every individual scraper, the group plans to introduce a single exclusion setting integrated into the standardized Robots Exclusion Protocol.

This report details the proposed robots.txt directive, the formal draft submitted to standards bodies, and vendor adoption timelines across the industry. The overview is designed for webmasters, system administrators, and content publishers managing scraping permissions. Coverage outlines Microsoft's separate AI Grounding controls for search queries, OpenAI's planned query safeguards, Google's deployment schedule, and Meta's current position.

Google: what actually changed

Google LLC, Microsoft Corp., Anthropic PBC, and OpenAI Group PBC have unified to address automated web scraping management. Currently, website owners rely on a text file titled robots.txt to communicate access rules to visiting crawlers. When an organization decides to prevent artificial intelligence developers from harvesting online content for model training, it must manually enter specific directives for each provider. If an artificial intelligence provider operates several crawlers to gather data, the webmaster must research and list an individual entry for every scraper deployed by that vendor.

The supported crawling standard simplifies maintenance by replacing multiple configurations with a single entry. Website operators will be able to block every automated scraper gathering generative artificial intelligence training data by adding one rule to their robots.txt file. This uniform exclusion applies specifically to generative training workloads. The coalition confirmed that the new setting will not apply to systems that perform exclusively non-generative tasks, ensuring that standard operational scrapers remain unaffected while preventing generative ingestion across digital archives.

Google: how it works

Directives in robots.txt files adhere to a standardized format known as the Robots Exclusion Protocol. Website operators historically used this protocol to manage search engine indexing, block crawlers from viewing specific webpages, or ask bots to limit visit frequency to conserve bandwidth. To incorporate their proposed opt-out mechanism into this format, Google, Microsoft, Anthropic, and OpenAI submitted a draft version of the standard to the Internet Engineering Task Force. This nonprofit body maintains the Robots Exclusion Protocol, making it the central organization for ratifying changes.

Beyond training exclusions, technology providers are building controls for real-time browsing. Microsoft stated in a blog post that it will provide a crawling setting called AI Grounding. This configuration addresses scenarios where an artificial intelligence model accesses a live webpage to answer a user question rather than collect training data. The AI Grounding control enables website operators to prevent Microsoft services from sending retrieval requests. The feature will also extend to third-party artificial intelligence tools that use Microsoft technologies to process prompts, while OpenAI plans similar controls.

Google: why it matters now

The rise of artificial intelligence scrapers created administrative friction for digital publishers. Historically, website administrators used robots.txt to coordinate with search engines, ensuring public pages appeared in search results without overloading server infrastructure. However, generative model developers deployed autonomous scrapers to collect training datasets, forcing webmasters to research vendor bot names and update configuration files repeatedly. Because providers often operate multiple scrapers, manual tracking became difficult. If an administrator omitted a specific scraper entry, that bot could collect proprietary site content without explicit permission.

Standardizing this rule through the Internet Engineering Task Force establishes a recognized industry baseline. By incorporating the standard into the Robots Exclusion Protocol, participating developers create an official mechanism for content governance. Furthermore, separating model training opt-outs from non-generative tasks ensures that basic web functionality remains protected. Routine data collection for traditional web utilities can continue uninterrupted under existing guidelines, preventing unintended outages in search index coverage while giving site owners clear control over whether their published content trains generative models.

Google: who is affected

The crawling standard directly affects website operators, system administrators, and content publishers. Organizations seeking to shield proprietary articles, forums, or media libraries from generative training will no longer need to track and update extensive user-agent lists. Applying a single robots.txt directive will signal an opt-out to all participating developers. However, administrators must note that the standard explicitly excludes non-generative scrapers, meaning tools performing indexing or standard web tasks will continue accessing public pages unless blocked by existing rules.

The policy also alters operational requirements for artificial intelligence vendors that crawl public web content. Google, Microsoft, Anthropic, and OpenAI have committed to the standard, requiring their automated collection infrastructure to parse and obey the single exclusion rule. However, adherence across the broader tech industry remains voluntary. While Meta Platforms Inc. expressed support for the idea behind the technology, the company has not yet committed to implementing it. Consequently, webmasters must recognize that scrapers from non-participating companies will not respect this unified setting until wider industry adoption occurs.

Google: what to watch

Technical teams must track staggered rollout schedules across participating organizations over the coming months. Microsoft and ChatGPT developer OpenAI plan to add support for the crawling standard in the coming months. Alongside this deployment, Microsoft will introduce its AI Grounding control, requiring administrators to review settings if they wish to block real-time query retrieval across native services and third-party tools using Microsoft technologies. OpenAI will similarly offer controls for real-time prompt retrieval, allowing webmasters to prepare for both training and inference directives.

Google plans to follow suit by adding support for the crawling standard sometime next year. Beyond the initial four backers, attention will focus on whether Meta Platforms Inc. formally commits to implementing the standard in its own scraping infrastructure. Industry observers will also monitor progress at the Internet Engineering Task Force as the organization evaluates the draft proposal for the Robots Exclusion Protocol. Formal adoption by the standards body will indicate whether this single-entry exclusion becomes an enduring standard across the web.