Cloudflare Requires AI Companies to Pay for Digital Content

A Turning Point in AI's Relationship with Digital Content

The internet has long served as an open pasture for AI companies, whose crawlers roam freely across its pages without paying anything to the publishers who created that content. Cloudflare has now decided to put an end to this reality, announcing a new policy that redraws the rules of engagement between major tech companies and digital content owners.

A Deadline of September 2026

Cloudflare has announced that as of September 15, 2026, its default settings will block all dual-purpose crawlers — those that combine traditional search indexing, model training, and intelligent agent operation simultaneously — from accessing any page that displays advertisements.

This decision will apply automatically to three categories:

• New customers joining Cloudflare after the policy takes effect.
• New websites created by existing customers.
• All existing customers on the free plan.

Website owners can manually adjust these settings if they wish to grant crawlers access, but the default state will be to block them.

Why Now?

Cloudflare co-founder and CEO Matthew Prince explained that the motivation behind this move is reaching a historic inflection point: for the first time, automated internet traffic has surpassed human traffic — a milestone that was expected to occur in 2026 but arrived ahead of schedule. In this context, Prince stated that "sustainability" has become the defining word that should govern the entire internet ecosystem.

Prince also noted that the world's largest search engine — in a clear reference to Google — enjoys twice the access of other AI companies, given that Google's mechanisms make it difficult for website owners to remain visible in search results without allowing their content to be used for AI purposes. Google, for its part, defended itself by affirming that it provides a "Google Extended" tool that allows website owners to exclude their content from training without affecting their visibility in search results.

From Pay-Per-Crawl to Pay-Per-Use

Cloudflare's initiative goes beyond simply blocking access; it encompasses the development of a comprehensive commercial framework that transforms digital content from a free commodity into a priceable asset. Having previously launched a "pay-per-crawl" model that charges AI crawlers a fee each time they fetch a page, the company is now announcing a transition toward a smarter model known as "pay-per-use."

This means publishers will not only be compensated when their content is fetched, but every time that content generates real value for AI systems — whether in the form of answers, search results, or recommendations.

This approach is also expected to reduce technical waste; Cloudflare's data reveals that more than 50% of AI crawler traffic is wasted re-fetching pages that have not changed.

Early Commercial Partnerships

To demonstrate the model's viability, Cloudflare announced partnerships with two companies in the first phase:

• Ceramic.ai: Pays publishers when their content appears in its AI-powered search results.
• You.com: Pays when accessing premium content from participating website owners.

Cloudflare notes that the model is flexible and can be adopted and adapted by other AI companies according to their own operational mechanisms.

Conclusion: Toward a More Equitable Digital Economy

Through these steps, Cloudflare is laying the groundwork for what could become a new equation on the internet: one that acknowledges that quality content has real value, and that AI companies benefiting from it should contribute to its sustainability. Whether major companies will voluntarily comply with this direction remains the most pressing question on the future of the internet.

✦ بقلم فريق دروب أيديا