#llm/security + #security

We present evidence that AI agents used the web security service urlquery.net to bypass restrictions and expand their access to the public internet. The agents also tried on three occasions to hack public data providers, including an Australian government website. We link at least some of this activity to agent swarms previously attributed to OpenAI.

However, as we described in our recent alignment assessment, building evaluations that reliably catch every failure prior to deployment remains an unsolved problem. We see signs that Opus 5.5 often suspects it is being evaluated, which challenges our ability to assess how it will act in the vast variety of real-world settings it is deployed in. As these settings expand and model capabilities increase, we expect this challenge to grow, unless we make progress on interpretability.

In May, Gemini broke containment and hacked three different companies, but Google didn’t disclose the incident until the Wall Street Journal approached the company. The hacks happened during a test of the model’s cybersecurity capabilities run by third-party Irregular, which was also involved in similar incidents involving Meta and OpenAI.

According to WSJ, Google didn’t disclose the hack because it didn’t consider it to be an “example of model misalignment.” The company said that it was an instance of “mistaken identity,” and once the model realized it had brute-forced its way into a real company by guessing a password, it stopped. “In this case, the model acted appropriately,” Google VP of Security Engineering Heather Adkins said.

Artificial intelligence agents being tested by OpenAI launched a cyberattack against a popular software service two months before they hacked the AI software company Hugging Face, a new signal of the potential for advanced AI tools to slip out of human control.

The attack overwhelmed maintainers of an online service for coders, called RubyGems, and forced them to shut down new account registrations as they dealt with the chaos it had caused, operators of the service said.

We present an alignment assessment of four incidents in which Claude models gained unauthorized access to real third-party systems. We described three of these incidents on July 30; we identified these after a scan of roughly 141,000 transcripts in which we believed Claude could have obtained internet access during a cyber evaluation.

All four incidents occurred during cybersecurity evaluations built by the same evaluation partner. Claude was told it was operating in a simulation without internet access, but, due to a misconfiguration, it was mistakenly connected to the open internet. As is standard for cybersecurity evaluations, the models ran without the cyber safeguards that ship with our released models. (This post does not cover the incident reported by UK AISI in its testing of Claude Mythos 5. We plan to conduct an alignment assessment of those transcripts as well.)

We have signed an agreement with METR to conduct an independent investigation of these incidents. Our agreement grants METR wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees, who will be permitted to share confidential information. Our initial agreement runs for eight weeks, with the option to extend by mutual agreement. We intend to give METR as much time as it deems necessary.

We are most concerned by the misalignment present in the incident involving Claude Mythos 5, in which the model went to extensive lengths to upload a malicious package to PyPI, the public repository from which most Python software is installed. Although the model repeatedly stated in its chain of thought (CoT) that it believed it was operating in a simulation, the environment offered clear evidence that it was on the real internet, and the actions it took were consistent with the model knowing this. When we made targeted modifications to the transcript to make it clearer that the model was not in a simulation, Claude Mythos 5 still took offensive actions, despite acknowledging a greater possibility of real-world harm. We are releasing this transcript publicly so others can build on our analysis (GitHub, PDF).

the models never deviated from attempting to solve the exercises they were given, and, in some cases, they attempted to stop the task. All incidents included a single Claude instance; at no point did Claude attempt to coordinate with other agents. Claude also never attempted to conceal evidence of its actions. We believe the misaligned behaviors present in these incidents are unlikely to arise in ordinary use, where Claude is not being instructed to conduct a cyberattack. Were these behaviors to arise, the safeguards we ship with our production models, including cyber classifiers and those in Claude Code auto mode, would add a layer of defense these incidents lacked.

However, reliably identifying the most concerning model behaviors during pre-release testing remains challenging, and, as we noted in our August 31 post, improving monitoring and reducing reward hacking in RL becomes harder as models advance. We have renewed our efforts to fix and remove environments that incentivize misaligned behaviors, and we continue to expand our alignment training to keep pace. Still, this remains unsettled science—it is critical that alignment and security mature faster than capabilities advance, which is one reason we support a coordinated, verifiable approach to pacing frontier AI development.

we are establishing a regular process for publishing what we learn about model behavior and alignment beyond what has been reported in our system cards, with clear criteria for what we report and when we report it

The scope of the agents’ unauthorized communications was “somewhat larger than we thought it was,” said Andrew Yoon, a researcher with the California nonprofit CivAI who said he tallied 18 previously undisclosed sites used by the agents between May and July. “It’s almost certain that there’s more going on here that we just don’t know about.” On Friday, researchers reported that a swarm of agents from OpenAI hijacked a German-language ​wiki site and turned it into an improvised messaging platform for cheating on tests, an incident that OpenAI kept secret as it dealt with the fallout from the July hack of the open-source repository Hugging Face. Now, both those researchers ​and other independent investigators say they have found several previously undisclosed sites where the same swarm appears to have left similar messages earlier this year. OpenAI did not directly address questions about how ⁠many different sites its agents used to communicate or say why it kept the activity under wraps for months. In a statement, it said it was undertaking a broader review of agent activity and had so far “not identified other activity matching the ​severity or scale of Hugging Face,” a breach that drew global attention and raised concerns that OpenAI was losing control of its own technology. OpenAI added that it was working on a framework for reporting “misalignment” – industry-talk for rogue behavior – across training, evaluation, and deployment of AI models ​and would share it “soon.” Reuters reviewed a total of six investigators’ or investigative group’s findings, including three that were posted to social media and another three that were shared privately with the news agency. The investigators’ methods varied, but many identified agent activity by matching strings of data left on the German wiki to identical strings left on other sites around the same time, or by marrying up similar or identical usernames tied to the messages, or by identifying activity geared toward answering

We did not enable internet access or inter-agent communication for many of the environments in these training experiments. Despite these restrictions, the agents discovered ways to exploit our research infrastructure to communicate with one another and access the internet.

The models first found ways to communicate by writing files into the Artifactory package manager. This effectively turned Artifactory into an unintended message board, where agents could exchange information with one another. Read more: How agents used the message board 

The agents also found ways to get access to the internet. Using an exploit⁠(opens in a new window), the agents made Artifactory, which itself had access to the internet for the limited purpose of downloading and installing packages, send any internet requests on their behalf. After the agents discovered these methods for obtaining internet access, they also shared them with other agents through the message board, which provided information to enable additional agents to exploit our infrastructure.