Cloudflare Hits the Kill Switch on AI Crawlers—And an Entire Industry Cheers

Cloudflare CEO Matthew Prince just made it official: Starting today, Cloudflare will block by default known AI bots from scraping sites unless given explicit permission—or unless they pay.

It’s not just a technical tweak. Cloudflare’s “Content Independence Day” marks a collective escalation. More than a million websites had already opted into blocking since last fall, but now it’s the default for every new Cloudflare domain. Furthermore, a roster of media heavyweights—from the Associated Press, Time, The Atlantic, and BuzzFeed to Reddit, Quora, and Universal Music Group—have signed on.

The movement has been well-received on social media, with investors and SEO experts praising the initiative.

“This is the way,” said said Ed Newton-Rex, former VP of audio at Stability AI; “Cloudflare solving problems again,” said pseudonymous trader and content creator Romano RNR. “It’ll be OK. A force is needed to tilt the balance back to where it once was,” said former Google and current SEO consultant Pedro Dias.

Of course, not everyone is convinced. The main argument against such a move is that it could hurt businesses not appearing on AI searches.

Or maybe not: Prince pointed out that a decade ago, Google crawled two pages for every one visit it sent to a publisher. Given that the number of people using the internet has skyrocketed to include most of the planet, publishers should benefit. But they haven’t, since in order to be “seen” on Google, publishers need to make their content easier for the search giant to summarize—the better to keep them within Google Search.

So today, for every “18 pages that Google takes from you, you get one visitor,” Prince said.

But the numbers for AI companies are exponentially more devastating. Said Prince: “What’s the ratio for OpenAI six months ago? 250 to 1. What is it today? 1,500 to 1.” And OpenAI is hardly the worst offender. Anthropic was “6,000 to 1 six months ago. What is it today? 60,0000” to 1, he said.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Prince argued. “I’m not going to sugarcoat it. If people don’t have incentive to create content, they’re not going to create content” he said separately. “We have to give that back or the internet dies,” Prince said.

“Pay-per-crawl”—or don’t crawl

To pressure AI companies into putting up or getting shut out, Cloudflare is pioneering a new approach to AI content access: a permissioned, pay-to-crawl marketplace. For the first time, publishers can set their own rates for AI bots—invoking the long-dormant HTTP 402 “payment required” code as a digital tollbooth.

Publishers can allow free access, charge for each crawl, or block them entirely. The company says this is only available for leading content creators in early access, but the plan is to scale the system.

The “pay-per-crawl” initiative arrives as some of the web’s biggest content providers seek leverage over AI companies, after years of having content ingested for free and monetized elsewhere. “When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Condé Nast CEO Roger Lynch.

This coalition-building, Prince believes, is essential.

“You cannot have a good sort of market without scarcity, which is why this needs to be a collective effort,” he said. He cautioned that licensing deals made today, if not coupled with real blocking, would be worse in the future.

Research confirms that the movement is more than symbolic. According to studies by Originality.AI and the Reuters Institute, 48% of the top global websites are already blocking AI crawlers. Last year, Cloudflare implemented an option to make it easy to block AI crawlers, and more than a million customers have exercised that option, the company said.

Now, new domains signing up with Cloudflare are asked upfront whether they want to allow AI crawlers, with blocking as the default. Existing customers can change their settings anytime.

Edited by Andrew Hayward