Cloudflare Launches AI Bot Marketplace to Charge for Scraping

Cloudflare Launches a New Marketplace for AI Content Access

Cloudflare, a leading cloud infrastructure provider serving 20% of all websites, has unveiled a novel marketplace designed to redefine the interaction between website owners and artificial intelligence companies. This initiative aims to empower publishers with increased control over their digital content.

Addressing the Rise of AI Crawlers

Over the past year, Cloudflare has introduced tools specifically designed to help publishers manage the increasing activity of AI crawlers. These include a simple, one-click solution to block all AI bots, alongside a comprehensive dashboard for monitoring AI crawler traffic to their sites.

In a 2024 discussion, Cloudflare CEO Matthew Prince explained to TechCrunch that these initial products were intended to establish a foundation for a new marketplace. This marketplace would facilitate the distribution of content to AI companies, with publishers receiving appropriate compensation.

Introducing "Pay per Crawl"

Cloudflare is now launching this marketplace, titled "Pay per Crawl," in a private beta phase. Participating website owners can selectively allow AI crawlers to access their content at a predetermined rate – essentially a micropayment for each instance of crawling. Alternatively, they can permit free access or completely block crawlers.

Cloudflare asserts that its tools will enable website owners to discern the purpose of crawling, whether it's for AI training data, inclusion in AI search results, or other applications.

A Potential Business Model for the AI Era

At a broader scale, Cloudflare’s marketplace represents a significant concept that could provide publishers with a viable business model in the evolving AI landscape. It also positions Cloudflare as a central player in this new ecosystem.

This launch occurs as news publishers grapple with fundamental questions about maintaining readership, particularly as traffic from Google Search declines and AI chatbots gain prominence.

Navigating the Challenges of AI and Content Licensing

The path forward for news publishers in the AI era remains uncertain. Some, like The New York Times, have initiated legal action against technology companies for utilizing news articles to train AI models without authorization.

Conversely, other publishers have established multi-year agreements to license their content for AI model training and to feature their content in AI chatbot responses. However, these licensing deals have largely been limited to larger publishers, and their revenue impact is still being evaluated.

Cloudflare’s goal is to establish a more sustainable system where publishers have the autonomy to set their own pricing.

Default Blocking of AI Crawlers for New Websites

The company also announced that new websites utilizing Cloudflare will, by default, block all AI crawlers. Website owners will need to explicitly grant permission for specific AI crawlers to access their site – a change designed to provide every new domain with “the default of control.”

Several prominent publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have committed to blocking AI crawlers by default, supporting Cloudflare’s vision of a “permission-based approach to crawling.”

The Shifting Dynamics of Online Publishing

The traditional business model relied upon by many publishers for decades is facing increasing instability. Historically, publishers have permitted Google to crawl their sites in exchange for referrals in Google Search, which drove traffic and, ultimately, advertising revenue.

However, recent data from Cloudflare indicates that publishers may be receiving less favorable terms in the AI era compared to the Google Search era. While some websites report ChatGPT as a traffic source, this is not a widespread trend.

Crawler Activity: A Comparative Analysis

Cloudflare’s data from June reveals a significant disparity in crawling activity. Google’s crawler scraped websites 14 times for every referral provided. In contrast, OpenAI’s crawler scraped websites 1,700 times for each referral, while Anthropic scraped websites 73,000 times for every referral.

Furthermore, OpenAI and Google are developing AI agents capable of visiting websites on behalf of users, gathering information, and delivering it directly. The widespread adoption of these tools could have substantial implications for publishers dependent on direct website visits.

The Potential of "Agentic" Paywalls

Cloudflare suggests that the full potential of Pay per Crawl may be realized in a future dominated by “agentic” technologies.

“Imagine an agentic paywall operating at the network edge, entirely programmatically. Consider requesting assistance from a sophisticated research program to synthesize the latest cancer research, a legal brief, or simply locate the best restaurant in Soho – and then providing that agent with a budget to acquire the most relevant content,” Cloudflare stated in a blog post.

Marketplace Mechanics and Future Considerations

To participate in Cloudflare’s experimental marketplace, both AI companies and publishers must have established Cloudflare accounts. Within their accounts, they can define the rates at which they are willing to buy and sell access to the publisher’s content. Cloudflare will act as the intermediary, processing payments from the AI company and distributing earnings to the publisher.

Currently, Pay per Crawl does not involve stablecoins or cryptocurrency, although Cloudflare is exploring the possibility of creating its own stablecoin or collaborating with existing providers.

Challenges and Opportunities

Cloudflare’s marketplace represents an ambitious vision for the future, requiring widespread adoption from both publishers and AI companies. There is no guarantee that publishers will secure favorable terms, and persuading AI firms to participate, given their current access to free content, may prove challenging.

Nevertheless, Cloudflare appears uniquely positioned to facilitate the creation of such a marketplace.

Update 2:50pm PT: This article has been updated to include the Cloudflare CEO’s comments on a podcast regarding stablecoins, and to correctly state OpenAI’s crawl-to-referral ratio in June, as reported by Cloudflare.