cloudflare ceo matthew prince is pushing uk regulator to unbundle google’s search and ai crawlers

Cloudflare Advocates for AI Regulation and Fair Competition

Following the launch of a marketplace enabling websites to monetize AI bot scraping, Cloudflare is actively seeking greater regulation within the artificial intelligence industry.

Matthew Prince, Cloudflare’s CEO, is currently in London to engage with the U.K.’s Competition and Markets Authority (CMA). He intends to propose more rigorous rules governing Google’s competitive position in the AI landscape, considering the company’s dominant role in search.

CMA's Designation of Google

The CMA recently assigned Google a special status within the search and advertising sectors, acknowledging its “substantial and entrenched” market position.

This designation empowers the regulator to implement stricter regulations extending beyond traditional search and advertising. These regulations could encompass areas such as Google’s AI Overviews, AI Mode, the Discover feed, Top Stories, and the News tab.

Cloudflare's Unique Position

Cloudflare believes it is well-positioned to offer recommendations due to its neutrality in the AI sector and its extensive network of relationships with AI companies.

“We don’t have a direct stake in this debate. We are not an AI company,” Prince stated at the Bloomberg Tech conference in London. “We aren’t a media publisher either, but we function as the network connecting them – with 80% of AI companies utilizing our services.”

The Issue of Fair Competition

Prince argues that Google should compete on a level playing field with other AI companies, a condition he believes is currently unmet.

He contends that Google leverages its existing web crawler to gather content not only for its search engine but also for its AI products and services, creating an unfair advantage.

(This refers to Googlebot, used for Search and AI features like AI Overviews. Google spokesperson Ned Adriance notes that website owners can prevent content use for AI training via Google Extended, without affecting search inclusion. However, some media outlets may prefer complete opt-outs, validating the core concern.)

Google's Content Access and Opt-Out Challenges

“Google asserts an inherent right to all global content, regardless of compensation, based on its 27 years of service,” Prince explained.

“They propose utilizing the same crawler for both search and AI systems, with a combined opt-out process. This presents a significant challenge for content creators.”

For many, particularly media organizations, forfeiting search visibility equates to a roughly 20% revenue loss, making a full opt-out impractical.

“Furthermore, blocking Google’s crawler also disables their ad safety team, disrupting advertising functionality across all platforms – an unacceptable outcome,” Prince added.

Bundled Crawler Advantages

By bundling its crawler, Google gains access to content that competitors like Anthropic, OpenAI, and Perplexity would otherwise need to acquire through payment.

“This situation risks handing the entire AI market to Google,” Prince warned.

Proposed Solution: Fostering Competition

Prince advocates for a highly competitive market, envisioning potentially thousands of AI companies vying to purchase content from numerous media businesses and millions of smaller enterprises.

He commends the U.K.’s CMA for recognizing Google’s unique advantage and designating it as a potential regulatory target.

Cloudflare has supplied the CMA with data illustrating the functionality of Google’s crawler and the difficulties other companies face in replicating its success.

Industry Consensus

Neil Vogel, CEO of People, Inc., the largest digital and print publisher in the U.S., recently echoed these concerns.

Vogel labeled Google a “bad actor,” stating that media companies are compelled to allow Google’s crawling due to the integrated nature of its crawlers.

Vogel, whose company has implemented Cloudflare’s solution to block unpaid AI crawlers, reports positive results, with ongoing negotiations with several major LLM providers.

Updated with Google comment.