Key Takeaways
Once a fun experiment, AI seems to have officially become a pillar in production infrastructure. The industry and its consumers have fast become dependent on generative and agentic A. But there’s one glaring issue which Akamai’s State of Inference 2026 report only confirms: The architecture needed for AI hasn’t yet caught up.
Inference was the natural next step that brought the internet from AI summaries to AI actions. Now sophisticated inference is expected. When we don’t get it, we become frustrated that we have to reprompt.
The report numbers validate it’s not unusual behavior anymore: 56% of organizations are using AI to personalize customer experiences, 52% for customer support, 51% for internal copilots.
Notice how these are all customer-facing, revenue-touching systems. With that comes new expectations. And the new bar is unforgiving.
Akamai reduced the responses into what they’re calling the three Rs of real-time AI: responsive (end-to-end latency under 250ms), reliable (performance maintained at the 99th percentile), and resilient (99.99% uptime).
These are the new SLAs — and while uptime never left the conversation, it sure got company. The problem is that 50% of organizations are already struggling to maintain that bar at peak demand.
“Most teams aren’t prepared for how unforgiving real-time AI actually is. Slow responses fundamentally break experiences for customers, which is a challenge a lot of companies underestimated,” said Ari Weil, Akamai’s VP of Product Marketing.
Real-Time AI Has an Infrastructure Problem
The value of distributed inference isn’t really in dispute: 98% of organizations recognize the business case, and 60% say proximity to users and decision points is important or critical. But nearly half are still running inference out of a single centralized cloud region, and 45% expect to still be doing that in a year or two.
“What catches them off guard is the operational reality,” Weil explained. “You’re no longer just connecting to or running a model; you’re operating a live system, and most of the tooling, architecture, and processes companies have been focused on for the past few years just weren’t built for that.”
Turns out it’s mainly tooling maturity (19%), cost uncertainty (16%), security concerns (14%). Building AI solutions requires a massive amount of compute, something only hyperscalers are well-positioned to continue owning.
“Hyperscalers will continue to dominate model training. AI factories will continue to be the powerhouses that fuel AI advances,” Weil said. “But inference is a different beat, more about speed, proximity, and consistency in real-world conditions than raw computing power. That’s where centralized models start to show limits.”
As Akamai emphasizes in its report, centralized systems weren’t built for real-time inference. But since the infrastructure’s not there, when inference gets slow or unreliable, teams try to keep the experience alive by rerouting and retrying (39%), downsizing (39%), or failing over (37%).
This is definitely not a long-term workaround, though.
So, What Should Hosts Do?
Akamai’s advice is about what you’d expect: distribute where latency matters, give teams better control over how workloads run, watch the cost of each request, avoid vendor lock-in, and don’t be afraid of using hybrid architecture when it makes sense.
For hosting providers, the governance conversation starts where the model providers’ ends. “Those questions belong to the model providers,” Weil said of training-time concerns like data licensing and consent, “and no amount of clever hosting can answer them after the fact.”
The bigger opportunity, though, is placement. Weil’s argument isn’t that edge providers should try to out-muscle hyperscalers on training, but that inference is a different problem entirely.
“The most logical outcome is a hybrid one where edge providers like Akamai extend the heavy lifting done by the hyperscalers out to the edge,” he said. “Distributed inference doesn’t solve AI governance. It solves the hosting-side governance problems that show up in every enterprise contract.”
In other words, the organizations that get this right will place workloads where they perform best: right provider, right region, right compute for the workload.
