Writer: Jordan Sprogis
Editor: Lillian Castro
Reviewer: Cristian Lopez
Key Takeaways
When HackerOne released its state of cybersecurity report last year, it presented quite a few concerning figures — one of which is that prompt injection attacks have increased 540% year over year.
Today, attackers still experiment with (and succeed at) manipulating AI systems through prompts at the fastest rate possible thus far.
Prompt injection is one of the most common ways attackers get into AI systems. And thanks to how much hosts are investing in AI, that means it’s now also a hosting problem.
What’s Actually Happening
Prompt injections occur when someone (a hacker, cyberattacker, botnet, whatever you want to call them) tricks the model into ignoring the original rules assigned and doing something else (like accessing private information).
The report urged hosting providers to go beyond prompt filtering and actually test whether their AI systems can be exploited the way bad actors are already doing it. And since hosts still have plenty to worry about, HackerOne launched Agentic Prompt Injection Testing to test exactly that.
“Hosting providers aren’t responsible for a customer’s prompts or model logic — but they play a decisive role in limiting the blast radius when AI apps are manipulated,” warns Sandeep Singh, a senior director at HackerOne.
Admittedly, it’s less about the tool itself and more about why something like this had to be built in the first place. It’s very much in line with zero-trust thinking: Don’t assume anything works the way it’s supposed to. Always double-check.
Now You Have to Test It
AI-powered systems and tools are so directly connected to data that it can take just one prompt mishap to cause an outage or breach.
It may be happening more than you think. IBM found that about 1 in 8 organizations (13%) have already experienced an AI-related data breach. More telling is that nearly all of them admitted they didn’t have proper AI access controls in place.
When breaches did occur, 60% led to compromised data, 31% caused operational disruption, and 23% caused financial losses.
Singh predicts auditors will expect hosts to actively test how AI systems react to being attacked, and not just rely on assuming everything will work as intended. As for how to do that, Singh has a few suggestions.
“Providers will either offer native capabilities or partner integrations that help customers test AI systems under adversarial conditions and produce clear evidence of what can actually be exploited before it becomes an incident,” he says.
It’s probably worth heeding that advice. Hosting providers are being pulled deeper into the AI security stack whether they like it or not. At this point, testing what a real attack looks like is really the only way to prepare.
