As AI Crawlers Skyrocket, Cloudflare and GoDaddy Move to Take Control of Traffic That Doesn’t Convert

Ask any hosting team what has changed the most about their customers’ traffic in the last two years, and the answer will probably be “bots.” And it’s unfortunately something far worse than the old-school scrapers that hunted for email addresses to sell and spam.

We’re talking about AI training crawlers, and these things are huge resource problems. They look like legitimate traffic but instead make hyperspeed repeated requests that puts strain on servers and uses resources without bringing in any real (i.e., human) visitors.

As of this morning, GoDaddy announced it’s officially integrating Cloudflare’s AI Crawl Control into its hosting platform, giving customers a dashboard interface that will let them allow, block, or set payment requirements for specific AI crawlers.

“The Internet is evolving into a high-velocity, AI-driven ecosystem, and that requires a new kind of transparent infrastructure,” said Cloudflare’s Stephanie Cohen. “We want to ensure that every creator has the tools to verify who is interacting with their site, while giving legitimate AI agents a secure, transparent way to participate in a thriving open web.”

The companies are also supporting two open standards: ANS and Web Bot Auth.

Agent Name Service (ANS) uses DNS and PKI to give AI agents a verifiable identity so platforms and operators can distinguish legitimate agents from everything else hitting their servers, including scrapers, bad actors, and cyberattackers.

It’s built on DNS, which GoDaddy, as the world’s largest domain registrar, arguably knows a thing or two about.

Web Bot Auth, introduced by Cloudflare last year, uses cryptographic signatures to verify bot and agent traffic. It’s paired with a Signature Agent Card — a structured identity declaration for developers — to create a verifiable chain of who the agent claims to be.

Cloudflare’s relationship with the hosting industry has always been interesting.

Its free tier converted customers who may have otherwise been paying for hosting-level performance and security features, like latency and DDoS protection. It basically worked as a layer of the hosting provider’s infrastructure; but this collaboration is entirely different because this time, it lives inside the host.

That also means that, for hosts competing in the SMB segment, GoDaddy now has a crawler management feature that most of them just…don’t.

Thing is, it’s not exactly cheap to build proprietary crawler tools. And redirecting customers to third-party solutions is becoming inadequate when almost everyone else has embedded solutions and features.

The hosting industry has seen this movie before: When Let’s Encrypt made TLSes free, hosts that were treating SSL as a premium add-on had to stop pretty much immediately. Otherwise, they’d risk losing customers to providers that offered it for, well, free.

It looks like ANS may have a similar trajectory. It runs on DNS infrastructure that registrars and hosting providers already use. And if this kind of verification for agents becomes the norm, hosts are going to have to change, too.

Of course, GoDaddy didn’t just decide to do this out of the goodness of its heart (though it’s very possible that’s a big part of it). With this collaboration, it’s positioning itself as an example of how AI agents are identified.

It’s a land grab, and a smart one at that.

The traditional web works because search engines send traffic to content creators, who then monetize that traffic.

AI answer engines just give the answer and move on, without sending users to the source, leaving creators with no traffic, no credit, and no way to monetize.

For hosting customers, their content is being scraped without sending anything back. For hosting providers, it means a problematic question: What happens to demand if creating that content stops being worth it?

Cloudflare’s AI Crawl Control lets site owners require payment before granting any access. ANS and Web Both Auth provides the identity verification layer that makes it totally enforceable.

Now, whether that actually happens is a whole other question…but ask the world’s largest CDN provider or the world’s largest registrar, and they’ll tell you it’s on the way.

“We move at the speed of the Internet,” said GoDaddy’s Jared Sine. “And we’re working with the broader industry to ensure the agentic open web does too.”