Cloudflare just put a price tag on the open web, and starting September 15 it will block AI crawlers by default on any page that runs ads.
On July 1, Cloudflare handed its customers a dashboard most of them didn't know they needed. It's called Attribution Business Insights, and it does one specific thing: it tells a publisher exactly how many times an AI company's crawler hit their site compared with how many actual readers that company sent back through a link. The numbers are ugly. According to Cloudflare's own blog post announcing the tool, the ratio runs from 118 crawls for every one human referral at the low end to nearly 50,000 crawls per referral at the worst offenders.
That dashboard was the appetizer. The real change lands September 15, when Cloudflare will default to blocking what it calls mixed-use crawlers, bots that blur search, AI agents and model training, on every page that carries advertising. Search crawlers still get in. Training and agent crawlers don't, unless a site owner flips the switch back on. The new default applies to every new domain that signs up with Cloudflare, every new site an existing customer adds, and all of its free-tier customers, according to Cloudflare's own changelog.
Cloudflare had already tried the toll-booth approach. Its Pay Per Crawl marketplace, launched last year, let publishers charge AI companies a fee every time a bot fetched a page. That model is now being replaced. Cloudflare found that more than half of AI crawler traffic was spent re-fetching pages that hadn't changed at all, paying for nothing. So it's rolling out Pay Per Use instead: publishers get paid when their content actually shows up inside a generated answer, not when a bot merely stops by. Cloudflare says it's testing this first with two partners, Ceramic.ai and You.com, where a publisher gets paid when their material surfaces in You.com's AI search results or gets pulled into Ceramic's outputs.
Matthew Prince, Cloudflare's co-founder and CEO, has been blunt about where this is headed. "Clearly it's going to be pay to crawl," he wrote, framing the shift as inevitable rather than aggressive. He's also pointed to a milestone that makes his case for him: bot traffic overtook human traffic on the internet for the first time this year, with agentic AI driving a 57.5% share, according to Cloudflare's own traffic data. When most of what hits your servers isn't a person, charging by the hit stops making sense. Charging by the value delivered starts to.
For AI labs, this is a cost problem wearing a policy costume. Training data has functioned as a free input for years, scraped at whatever volume a crawler could manage, with no metering and no bill attached. Cloudflare's own figures suggest why that free ride mattered: a crawler running at 50,000 hits per referral was pulling enormous volumes of text for a sliver of the traffic it ever sent back. Multiply that gap across every major model provider and you start to see how much of frontier AI's data pipeline has leaned on scraping nobody had to pay for.
Cloudflare isn't the only company sitting between publishers and crawlers. Fastly and Akamai run large chunks of that same infrastructure, and neither has announced anything close to a September 15 default block. Fastly's bot handling still leans on manual rules that customers configure themselves. Akamai's Bot Manager sorts crawlers by behavior rather than drawing a hard line on ad-supported pages. Neither company has said whether it will match Cloudflare's move, but Cloudflare sits in front of enough of the internet's traffic that its defaults have a habit of becoming everyone else's expectations, whether or not competitors formally follow.
Nothing forces OpenAI, Google or Anthropic to pay a cent more than they do today. What changes on September 15 is the default. Getting the data now means either paying for it through Pay Per Use or asking a publisher to flip the switch back to open. For an industry that trained its models on a web that never charged for entry, that's a genuinely new line item, and one nobody has priced into a training budget yet.
Also read: Kevin Warsh Just Put Marc Andreessen In The Room Where Fed Policy Gets Made • Nandan Nilekani Steps Back From Fundamentum to Bet Bigger on Fund III • Nvidia Backs Paris Startup Gradium, Pushing Its Seed Round Past $100 Million
