Cloudflare and GoDaddy are building new infrastructure to let website owners control, block, or charge AI bots that scrape content without sending traffic back, aiming to fix a broken online economic model.
The internet's oldest business model is quietly falling apart, and two of its biggest infrastructure players have decided that enough is enough. Cloudflare, which handles traffic for roughly 20 percent of all websites, and GoDaddy, the world's largest domain registrar, are partnering to give site owners something they have lacked since the generative AI boom began: real control over who gets to harvest their content and on what terms.
At the center of the deal is Cloudflare's AI Crawl Control, a tool that GoDaddy will integrate directly into its hosting platform. For millions of small businesses, bloggers, and independent publishers who lack engineering teams, the feature offers a simple dashboard to decide whether AI bots from companies like OpenAI, Anthropic, and Google can access their pages. Site owners can grant access, block crawlers entirely, or, crucially, set the stage for charging AI companies for the data they extract.
The timing is not accidental. As Business Insider reported, the partnership is a direct response to a structural shift in how people find information online. For two decades, the web operated on a mutually beneficial exchange: search engines indexed content, users clicked through to the source, and publishers earned revenue through advertising, subscriptions, or commerce. Generative AI has disrupted that loop almost overnight. When a chatbot delivers a synthesized answer pulled from dozens of sources, the original creators see no traffic, no impressions, and no income.
The numbers paint a stark picture. Cloudflare's own data has shown that AI crawlers now account for a significant and growing share of all bot traffic on the internet, yet they return a fraction of the visits that traditional search engines like Google or Bing historically delivered. Major publishers including The New York Times, Condé Nast, and Reddit have all taken steps to restrict AI access or negotiate licensing deals, but those actions require resources that most small operators simply do not have.
This is where the combined reach of Cloudflare and GoDaddy becomes genuinely consequential. Together, these two companies touch a massive portion of the web. GoDaddy alone manages over 80 million registered domains. By embedding crawl controls directly into a hosting environment that small businesses already use daily, the partnership could democratize a capability that was previously limited to well-funded media companies with technical expertise.
Beyond the crawl controls, the two firms are also backing new technical standards designed to bring order to what has been a lawless environment. The proposed Agent Name Service and Web Bot Auth protocols would verify the identity of AI agents and create a transparent record of their behavior. Right now, many AI crawlers mask their identity or rotate through IP addresses to evade detection. Standardized authentication would make it far harder to scrape content anonymously.
What a Permission-Based Web Means for Business
The broader ambition here is nothing less than establishing a permission-based framework for the AI-driven internet. Instead of the current free-for-all, where bots vacuum up content with minimal oversight, the vision is a system where creators can see exactly who is accessing their work, set terms for that access, and receive compensation when AI models generate value from their data.
This builds on Cloudflare's earlier decision to start blocking AI crawlers by default across its network, a move that sent a clear signal about where the infrastructure giant stands. The company has also been developing tools that allow publishers to set prices for AI access to their data, essentially turning web content into a licensable asset rather than a freely available resource.
For startups and small businesses watching their organic traffic decline as AI answer engines absorb more queries, the practical takeaway is straightforward: the tools to fight back are finally becoming accessible. If you run a site on GoDaddy or use Cloudflare's services, expect to see new options in your dashboard soon that let you audit AI bot activity and set access policies without writing a single line of code.
The stakes extend well beyond any single publisher or platform. If AI systems continue extracting value from the web without sending meaningful traffic or revenue back to creators, the incentive to produce original, high-quality content will steadily erode. A web without fresh content is a web that AI models themselves will eventually struggle to learn from, creating a vicious cycle that harms everyone in the ecosystem.
What to watch next is whether the major AI companies embrace these standards voluntarily or whether they will require regulatory pressure to come to the table. The European Union's AI Act and proposed copyright frameworks in several countries are already pushing in this direction. If Cloudflare and GoDaddy can prove that a permission-based model works at scale, it could reshape the economics of the AI internet before regulators even finish writing the rules.
