Cloudflare tightens AI crawler controls for publishers

Cloudflare has introduced new controls, analytics and payment arrangements for how AI crawlers access online content. The changes target publishers and AI companies that use the web for search, training and agent services.

It is changing default settings for some websites, expanding reporting on AI bot activity and shifting its compensation model from charging for crawling to charging when content is used in AI results. Cloudflare is also working with You.com and Ceramic.ai on the payment system.

The measures come as automated traffic now accounts for more than half of all web requests, according to Cloudflare. That shift is changing how websites are discovered and increasing pressure on publishers that rely on advertising or subscriptions.

New defaults

A central change affects websites where Cloudflare detects that the business model relies on advertising. For those pages, if a crawler does not separate search discovery from AI training and agent use, the default setting will block that crawler while still allowing search access.

Customers will be able to change those settings in their dashboards. After a testing and feedback period, Cloudflare plans to apply the new defaults to new customers, new sites for existing customers, and existing free customers that have not altered their settings.

Cloudflare argues that mixed-use crawlers force publishers into a trade-off between visibility in AI-driven search and giving up content for training or agent use without payment. Some AI companies have already distinguished their bots by purpose, making it easier for site owners to decide what to allow.

Cloudflare also said the largest search engine currently has access to roughly twice as much information as leading AI companies because site owners can struggle to remain discoverable without also permitting AI use.

Crawl waste

Alongside the policy changes, Cloudflare published data on how AI crawlers use websites. More than 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages, it said.

According to Cloudflare, that pattern increases bandwidth costs for publishers and computing costs for AI companies without improving answers. It is testing signals that indicate whether a page has changed and therefore needs to be fetched again.

The aim is to reduce repeated requests for the same content while preserving freshness in search and answer services. Cloudflare said it is testing those signals with AI companies before a wider rollout.

Payments and data

Another part of the launch is a new business dashboard for publishers. Cloudflare said the Attribution Business Insights dashboard is designed to show how AI bots consume content, which pages attract the most bot attention and how much human traffic AI services send back to websites.

Cloudflare framed that as a way to address a long-running information gap for publishers negotiating with AI firms. More than 50 major content licensing agreements have been signed between publishers and AI platforms over the past year, it said.

Cloudflare is also changing its commercial model. Its earlier Pay Per Crawl system charged AI companies to fetch content, but the new Pay Per Use model is intended to pay publishers when their content appears in results or otherwise creates value.

With Ceramic.ai, participating publishers will be paid when their content appears in that company's AI search results, with data returned on queries, citations and ranking, Cloudflare said. In You.com's case, the model will allow an agent to pay on demand for a specific piece of premium content.

Matthew Prince, Chief Executive Officer of Cloudflare, set out the company's position on the changes.

"Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem. Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training," said Prince.

Partners and publishing platforms also backed the approach, particularly where it distinguishes between discoverability and training access.

"To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation. Cloudflare allows us to easily and programmatically scale our operations. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results," said Anna Patterson, Founder and Chief Executive Officer of Ceramic.ai.

Patreon also linked the changes to creator control over AI access.

"As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies. On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used. That's why we're building on our existing work with Cloudflare to block known AI training crawlers at the network level across Patreon, while still allowing the crawlers that help creators get discovered and grow their businesses through search," said Drew Rowny, Senior Vice President of Product.