Key Takeaways
- When Cloudflare began blocking AI bots by default, Daphne Monro says hosts were suddenly facing the same choice.
- In an agent-first web, hosts must choose between allowing, disallowing, or controlling crawler access.
- And yet, even in a web that's operated by bots, someone still has to create something worth crawling.
When Cloudflare announced it was going to begin blocking bot crawlers by default, it raised a new dilemma for hosting providers: Should they follow suit or not?
“That caused a lot of concerns from customers wondering what’s the right thing to do,” said Daphne Monro, head of website and content at Hosting.com.
It’s a double-edged sword. The very traffic that helps clients surface in AI overviews and GPT conversations is the same traffic that’s costing providers (and sometimes their clients) an arm and a leg.
Content will likely never be outdated, but discoverability — today’s buzzword for the quality of how your clients’ sites are found — is absolutely dictated by how cleanly and quickly agentic AI can understand it.
At the same time, hosts are debating whether enabling AI bot controls actually hurts search traffic. It may save on performance and cost, but it may also mean sacrificing clients’ sites visibility.
Option #1: Block Bots and Stop Straining Your Bandwidth
Since crawlers now account for around 20% of verified bot traffic, people from all over are reporting millions of automated requests in short periods — in some cases, totally eclipsing human visitors altogether.
AI crawlers can consume enough bandwidth and CPU to skew analytics. Vercel itself logged more than 500,000 automated requests in a single month.
On shared infra, that’s the stuff of nightmares.
Monro argues the debate isn’t really about content at all. It’s about infrastructure and the kinds of choices providers are willing to make.
“Your hosting infrastructure directly affects how these agents and bots and crawlers are engaging with your site and how quickly they can access the information,” she said.
Blocking bots can pretty much guarantee stabilized performance and cloud costs, which is ideal for shared environments. That’s all good until clients stop showing up in AI search results.
Option #2: Allow Bots and Feed the Machines What They Need
Being crawlable really isn’t optional anymore. If you want your clients’ sites to be seen, of course.
“We highly recommend ensuring that your content can be crawled if being visible to a wider audience is where you’re trying to be — which most business owners are,” Monro emphasized.
Many internet users have already been relying on LLM interfaces for searches and answers instead of clicking through search results.
A few years ago, Google executive Prabhakar Raghavan testified that nearly 40% of young users turn to TikTok or Instagram for search-related queries (like restaurant discovery) instead of Google Search or Maps. And Gartner projects that by the end of this year, traditional search engine volume will drop 25% as users move to AI chatbots and virtual agents.
Regardless of whether that prediction will be 100% accurate, there’s no argument that users are relying on asking systems for answers instead of clicking through results.
And that means those agents have to be able to find the sites that have the information being requested.
“Structure your content by making sure that you have the right schema, product feeds, APIs, and accessibility, so that your content can be read by these bots, and served up correctly,” Monro added.
Option #3: Control Bots and Protect Your Infrastructure
The web hosting industry seems to love believing every new change in the market or trends is unprecedented. But it rarely is. Even with how technologically advanced we’ve become, we’ve actually been here before.
The early ‘90s web revolved around uptime, which ultimately set the long-lived baseline for being “web-ready” for years to come. By the mid-2000s, getting your site found became a game of algorithm and a better chance of you mastering SEO tactics like backlinking.
Nowadays, hosts are expected to do so much more than provide the infrastructure, but also act as partners and advisors. And when new technologies come about, clients expect their hosts to be there, waiting with open arms.
Monitoring crawl statistics can be helpful, especially with Google Search Console, which has an entire Crawl Stats report that logs how many requests Google made to your site over time, how often, and how your server responded. Googlebot, for example, adjusts its crawl rate based on the server.
Plus, schema markup could result in a 20-30% higher click-through rates. One test, in particular, says that adding product schema coincided with 30% surges in organic traffic for eCommerce.
But in the specific case of AI agents, that may look a bit like policy. No, not to the point of bureaucracy — but internal policies.
“Bots don’t browse, so you have to think of the things they can execute,” Monro said.
That may start with the basics, like refining robots.txt and experimenting with LLMs.txt to offer cleaner summaries for AI agents.
Though new, Cloudflare released a pay-per-crawl for high-volume access, which is available to anyone on Cloudflare’s network. Or you could opt for stronger WAF rules that distinguish verified crawlers from spambots and rate limiting to prevent burst traffic for shared environments.
“We do recommend always ensuring that you have that extra buffer room. If you’re better optimized, you are more likely to be shown up in search,” Monro told us.
The flip side of the coin is that machines, like humans, have limits to their patience. If your protocols force them to jump through hoops, then the crawler will likely bypass the site altogether.
“They [have to] engage with your site fast enough. It’s very expensive to crawl the web,” Monro said. “By being quick, you get your information out correctly, efficiently, and save time and money.”
Surviving In an Agent-First Digital World
All of this — blocking, allowing, controlling — assumes humans are still the primary audience. But Monro isn’t so sure this will remain the case.
“I think websites are going to be obsolete from a human perspective,” she said.
It’s a bold statement, especially coming from a head of content. But she’s not suggesting original sites will become obsolete. In fact, Monro has one piece of advice for anyone who’s listening.
“Just remember to be human. I think that everybody needs to remember that people want to hear the friction, the counter arguments,” she said. “That’s what makes content human and engaging.”
Because even if discovery becomes agent-driven, someone still has to produce something worth crawling.
