Key Takeaways
- AI bots now crawl up to 85% of sites daily, and Hostinger’s new AI Audit tool gives clients and resellers the ability to track, block, or allow them access.
- AI crawlers are driving up bandwidth costs without clicks or revenue, which is what pushed Hostinger to give clients new tools for control.
- Simple protocols exist that can give providers the power to manage how AI crawlers interact with client sites.
One in three of the busiest web crawlers now belong to AI companies — and according to Hostinger, they’re hitting as many as 85% of sites daily. In response, Hostinger launched a new tool, AI Audit, which allows its clients to see which crawlers visit their site, track what they do, and block or allow certain bots.
“This is our first version of the AI audit. Early usage shows tens of thousands of clients already monitor crawler activity, and hundreds already use blocking,” Hostinger told HostingAdvice. “We’ll continue tracking AI crawlers and client behavior to improve flexibility and protection.”
More specifically, Hostinger’s findings show that:
- Googlebot, Metaexternal-agent, and ChatGPT crawlers make up 60-85% of site visits per day
- Applebot, TikTok, and Petalsearch make up 20-25%
- Claude, Amazonbot, and Perplexity make up 2-5%
Crawlers are taking up massive amounts of bandwidth with little return on click-throughs and traffic (if any). And for resellers that depend on Hostinger’s infrastructure, this is game-changing news because now they’ll be able to offer the same thing to their clients.
AI Crawlers Are Reshaping Search Economics
AI crawlers are emerging as the next evolution of traditional search — just as SEO once was.
For years, sites were built with SEO in mind, which focused on optimizing keywords, structure, and backlinks to rank higher. Eventually, Google Search relied on Core Web Vitals, which prioritized more usefulness to the content and the overall user experience.
Now, generative engine optimization (GEO) adds another layer that makes sure AI crawlers can parse and cite what’s considered high-value, relevant site content.
The issue is that AI crawlers don’t bring traffic, ad impressions, or affiliate conversions. Instead, they pull content and surface it inside third-party platforms like ChatGPT or Google’s AI Overview.
Companies have already reported thousands in extra bandwidth costs from crawlers hitting their pages nonstop. And yet, they’re seeing no revenue, since those visits don’t bring real traffic. And when traffic falls, clients turn to their host for answers.
That’s a serious issue, said Giedrius Zakaitis, Hostinger’s Chief Product and Technology Officer.
“On one side, businesses see an opportunity and even use [GEO] to gain visibility in AI responses,” said Zakaitis. “Meanwhile, for those monetizing on visits, this shift creates challenges. If an AI tool provides the full answer, only a few readers will click through.”
Hosts Push Back With AI Audit Tools
Hostinger isn’t alone in creating an AI auditor.
Cloudflare recently rolled out its own AI scraper mitigation system that lets sites on its CDN block or allow AI bots by default — and even charge them for access. Similar to Hostinger’s AI Audit, it builds on protocols such as robots.txt and llms.txt, but the difference is Cloudflare enforces them directly at the network layer instead of at the hosting layer.
Those protocols are simple enough for hosting providers to deploy on behalf of their clients, giving sites more control over unwanted crawlers, bandwidth use, and content protection.
Which crawlers to allow is entirely up to the site owner, Hostinger told HostingAdvice.
“Some may benefit from AI tools like ChatGPT or Perplexity, while others may block them but allow other AI crawlers that add value,” they said. “If a specific AI tool misuses content, blocking just that crawler can protect their site while maintaining visibility in other AI tools.”
Misusing content isn’t uncommon. Some bots ignore llms.txt and robots.txt permissions, while others crawl excessively and drive up bandwidth costs, as aforementioned.
In many cases, content is scraped to train AI models — leaving site owners in the dark about where their material is used. For example, The New York Times sued OpenAI and Microsoft, accusing them of using millions of its articles without permission to train their LLMs.
Still, for site owners looking to stay visible in the next generation of search, the shift isn’t without opportunity.
“For those who want to increase visibility in AI-generated responses, we recommend adopting GEO strategies: create well-structured content, improve website loading speed, and add an llms.txt file to make it easier for AI tools to navigate their site,” Hostinger said.
What SEO looks for isn’t so far from GEO, and now, the job for providers is to make sure crawlers see what’s worth seeing.
