Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare just released a new deadline for the AI ​​industry to separate web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare’s default settings will block “multi-use” crawlers from any pages that host ads, the company announced Wednesday.

This means that crawlers that combine search, proxy use, and training will be prevented from crawling these sites by default, unless the site owner adjusts the settings otherwise. The company says these default settings changes will apply to new Cloudflare customers, new sites created by existing customers, and all existing free customers.

The move could impact how AI model providers are able to access web content for training purposes and to help power their proxy services.

Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protection from giving away their intellectual property for free.

Cloudflare specifically describes the “world’s largest search engine” (obviously a Google reference!) as having access to “roughly 2 times more information” than other AI companies because the search giant makes it difficult for customers to remain discoverable without their AI.

Google has disputed this generalization in the past, noting that it provides a bot called Google Extended It allows site owners to opt out of the use of their content for training and AI products and services such as Gemini Apps and Vertex API. Its use does not affect the site’s listing in Google search. However, the tech giant’s flagship Googlebot crawls search, including artificial intelligence features like AI Overviews and AI Mode.

“Now that the majority of Internet traffic is non-human, we must move forward and act faster so that a sustainable ecosystem can emerge,” Matthew Prince, co-founder and CEO of Cloudflare, said in announcing the news, referring to the latest achievement. Where bots have overtaken human internet traffic For the first time. This shift was not expected to happen until next year.

“Cloudflare’s new tools and partnerships give website owners more visibility and business opportunities and benefit AI companies with bots with clear and transparent intent. We hope the proposed default changes will encourage versatile crawlers to separate research from agent use and training,” Prince said.

While Cloudflare offers a number of products to help users Launching their own artificial intelligence systemsthe company also released a set of tools to give publishers more control over their content in the age of artificial intelligence. In recent years, Cloudflare Launching tools to combat artificial intelligence botsincluding A A marketplace that allows websites to charge AI bots for scrapingit’s called “pay per crawl”.

The latter is now also evolving into “pay-per-use,” which will allow publishers to charge AI companies when their content creates value, not just when it is fetched, the company said.

The change can also help conserve bandwidth for publishers and compute resources for AI model providers, as Cloudflare data indicates that more than 50% of crawl traffic from AI crawlers is spent refetching unchanged pages.

To do this, Cloudflare is initially working with two partners, Ceramic.ai and You.com. When a publisher signs up, they get paid when their content appears in Ceramic’s AI search results or when You.com accesses a piece of their premium content.

Cloudflare says other AI companies can customize this model for how they work.