The hosting service aims to grant websites greater authority over the usage of their content by AI firms.
Cloudflare has unveiled its intention to automatically block hybrid web crawlers that perform dual roles: indexing websites for search engines while also functioning as AI agents and trainers. Previously, the company provided its clients with the option to prevent such crawlers from scraping their sites for AI chatbot purposes; however, it is now adopting a more proactive default stance.
“As non-human traffic increasingly dominates the Internet, it is imperative that we take more decisive action to foster a sustainable ecosystem,” stated Matthew Prince, CEO and co-founder of Cloudflare. “Our newly developed tools and partnerships aim to increase visibility and economic prospects for website owners while also benefiting AI firms that operate bots transparently. We aspire to encourage mixed-use crawlers to distinguish between indexing for search engines and utilizing content for AI training.”
Website traffic formerly indicated genuine human engagement with ads or subscriptions, but the rise of AI models that can access sites on behalf of users has disrupted this paradigm. With its new strategy, Cloudflare seeks to reestablish balance, making it equitable for both AI entities and website operators.
Effective September 15, 2026, new users and existing customers launching new sites on Cloudflare will have their accounts set by default to allow search functionalities while blocking training and agent use on ad-supported pages. Hybrid crawlers that do not offer website owners an option for AI content use will face automatic blocking on pages with advertisements. This change will also apply to free account users unless they opt out before the September deadline.
In conjunction with these modifications, Cloudflare will update its Pay Per Crawl feature introduced in 2025, which permitted websites to forbid AI crawlers from accessing their content unless they paid. The revamped feature will be termed Pay Per Use, where compensation for site owners will be based on their content’s appearance in AI chatbot responses rather than on crawling activity. While initial partnerships have been formed with Ceramic.AI and You.com, Cloudflare aims to attract additional AI firms to participate as customers opt in.
In an effort to promote fairness in the dynamic between websites and AI companies, Cloudflare seems to also be casting attention toward Google. The announcement indicates that the leading search engine has access to significantly more information than top AI competitors, which complicates the discoverability of sites without their content being used for AI. Google’s primary crawler, Googlebot, serves dual purposes: indexing websites for its search features and collecting data to enhance AI capabilities. Although Google provides an alternative crawler, Google-Extended, for traditional search indexing, publishers wanting inclusion in AI result modes face limitations if they prefer not to contribute their content for training Google’s models. Cloudflare’s new directive may pressure Google and similar firms with hybrid crawlers to adapt their operations.
