Some of the biggest names in AI are aligning around a more publisher-friendly way to manage web scraping. OpenAI, Google, Microsoft, and Anthropic say they will adopt a proposed standard that allows websites to express their preferences about AI training through a shared signal.
The approach, called a policy token, would be placed inside a site’s robots.txt file — the long-running system websites use to communicate with crawlers. Instead of configuring separate opt-out rules for each AI company, publishers could use one common mechanism to communicate their choices across multiple platforms.
Why this matters
This is a meaningful step toward a healthier AI ecosystem: one where developers can keep building powerful systems while publishers gain clearer, simpler tools for consent and control. By working through the Internet Engineering Task Force, the proposal also has a pathway toward becoming a more widely recognized internet standard.
- Simpler controls: Publishers can manage AI training preferences in one place.
- Broader consistency: Major AI companies backing the same signal reduces fragmentation.
- More trust: Clearer opt-out tools can improve relationships between AI labs and content owners.
While the standard is still emerging, support from multiple leading AI organizations is a positive sign. It shows that the industry is moving toward more transparent and interoperable norms for how AI systems interact with the open web.