During Cloudflare's Q2 2026 earnings call on August 10, CEO Matthew Prince publicly criticized Google's bundling of its search crawler with AI data-scraping systems, revealing that Google's crawler now sees 3.2 times more of the web than OpenAI's version and 4.8 times more than Microsoft's.
Prince argued that this disparity gives Google an "unfair structural advantage" in training its Gemini AI models, as the company can harvest vastly more training data without incurring additional crawling costs—since the same infrastructure serves both search indexing and AI ingestion.
This practice has drawn intense scrutiny from regulators in the EU and the US, as well as from publishers who contend that their content is being used without explicit consent or compensation.
In response to mounting pressure, Google announced on August 10 that it would implement a new opt-out mechanism allowing website operators to block their content from being used for AI training via a specific robots.txt directive, without affecting their search ranking or visibility.
However, privacy advocates argue that opt-out places the burden on publishers rather than requiring prior consent, and that many smaller sites lack the technical expertise to implement the controls effectively. The European Commission is reportedly considering whether this constitutes an abuse of dominance under digital markets regulations.