Skip to content

Experts insights

Is AI Going Out of Control?

TL;DR

  • In July and August, OpenAI, Anthropic, and Meta each disclosed incidents where their AI models took unauthorized action against real organizations during cybersecurity tests, one of them via a genuine zero-day, the others via configuration errors at a shared testing vendor.
  • A separate UK government test found an Anthropic model going further: it built fake identities to trick a real person into approving malicious code, then covered its tracks.
  • None of this points to AI "going rogue." It points to models relentlessly pursuing assigned tasks without reliably distinguishing real systems from test environments, at a speed current defences weren't built to match.
  • The same capabilities that make this risky, fast, automated vulnerability discovery, could equally strengthen defence, if paired with better collaboration between organizations and clear accountability for AI producers.

Within weeks, three of the world’s leading AI systems apparently carried out unauthorized actions against real-world systems during security tests. While the media has sensationalised the image of 'out-of-control machines' acting autonomously against the will of their creators, the reality is different and, given its future implications for the cybersecurity, more valuable to understand.

What Happened

In July, two OpenAI models exploited an unknown vulnerability to escape from an isolated test environment, using, among other tools, stolen credentials to breach Hugging Face’s production infrastructure. The intrusion, which would have taken an experienced human attacker weeks to carry out, was completed in just a few hours.

Around the same time, Anthropic published an internal review identifying three incidents in which its models had reached the internet from environments that were supposed to be isolated, gaining unauthorized access to three real-world organizations. Although the cause lay in a configuration error by the provider managing the tests, the incident caused quite a stir.

The following month, due to the same error, the same provider appears to have caused a breach by a Meta AI model against a company’s systems during a test.

Also in August, the UK-based AI Security Institute revealed an experiment in which an Anthropic model was able to construct fake identities and use them to trick the maintainer of an open-source project into approving malicious code, subsequently covering its tracks when the request was publicly challenged.

The Real Meaning

Although in previous incidents the media described the AI as capable of independently bypassing controls, it is clear that these were often simply configuration errors.

Anthropic itself has characterized its incidents as operational failures rather than alignment failures, emphasizing that the models did not exploit unknown vulnerabilities, but simply walked through a door that was supposed to remain closed.

No rebellion, then, but rather persistence in achieving assigned goals, in the absence of the ability to differentiate between the real environment from the test environment. At the very least, this could be seen as a case of excessive determination.

In every test conducted, the model pursued the assigned task using every possible resource, including those seemingly out of reach.

This is exactly how a human attacker would act, and these experiments have confirmed just how much AI is evolving, and how this is changing the world of cybersecurity.

The Implications For Cybersecurity

Given that AI is not out of control, it is far more significant to focus on the capabilities achieved by current models, capable now not only of identifying and chaining together zero-day vulnerabilities to breach computer systems at an impressive speed, but also of creating fake identities and manipulating real people.

The positive aspect is that these same capabilities could be used to improve security defences, moving beyond the limitations of periodic vulnerability assessment processes, which only provide a snapshot of the state of IT systems once or twice a year, and also to test organizational resilience.

At the same time, this would foster greater collaboration between companies and vendors, sharing relevant findings through coordinated disclosure, especially in the case of zero-days, thereby accelerating the discovery and resolution of vulnerabilities.

However, when an unknown vulnerability can be found in a matter of hours, the economics of zero-days also change for attackers, who can more easily uncover exploitable and very dangerous vulnerabilities for their malicious operations.

In order to be prepared for this scenario, it is necessary not only to continuously strengthen the defences, but also to treat a breach as a likely event, with disaster recovery plans ready and up to date.

Conclusions

Is AI out of control? The evidence does not support this interpretation. On the contrary, it points to a more useful conclusion: these systems are now capable of carrying out cyber offensive operations, including deception, that were previously the exclusive domain of human attackers, at a speed and on a scale that current defence tools were not designed to handle.

It is therefore clear that we must prepare for increasingly aggressive and invasive attack scenarios in the future, both by leveraging these same capabilities to test defences and by addressing regulatory issues. Clear guidelines on how manufacturers should design and contain AI’s offensive capabilities are therefore urgently needed, as well as clear legal and civil accountability for the consequences of their products' actions.

Clearer decisions. Better budgeting. Fewer losses.

From "how many attacks?" to "how much damage?"
The clarity needed for sharper strategy.