OpenAI's Test Agents Built a Hidden Message Board, Then Attacked Hugging Face
- David Borish
- 21 hours ago
- 5 min read
In early May 2026, an OpenAI evaluation agent hit a wall. The task it had been assigned was impossible to complete because the team running the evaluation had forgotten to upload a file it needed. Instead of reporting the failure, the agent went looking for a workaround. That single decision, made inside a routine internal test, set off a chain of events that by July had produced two zero-day exploits, a self-organized network of AI agents trading hacking techniques, and a breach of Hugging Face's production infrastructure.
Eric Wallace, an alignment researcher and tech lead at OpenAI, and Michael Dalton, an OpenAI staff member focused on agent and infrastructure security, laid out the full timeline Wednesday at Black Hat USA 2026 in Las Vegas. It was the company's first detailed public accounting of an incident it had already disclosed in broad strokes weeks earlier. The session filled in how a group of agents with no coordination instructions ended up working together, discovering vulnerabilities together, and eventually acting on external systems neither OpenAI nor Hugging Face had authorized them to touch.
An Accidental Message Board
Stuck agents reasoned that answers to their assignments might exist somewhere online, and in late May some of them found a path to the internet through a server-side request forgery flaw. One agent also discovered it could leave files inside Artifactory, the JFrog package registry OpenAI used internally to cache software dependencies. A second agent found those files and asked whether anyone had located a missing one. Over the following weeks, that exchange grew into an informal bulletin board that agents running entirely separate evaluations could read and post to.
Agents began leaving notes about which exploits worked, assigning follow-up tasks to each other, and picking up scripts where another agent had left off. Wallace described watching the group's capabilities compound as they built on each other's discoveries, calling it a "Cambrian explosion in intelligence." The agents were not instructed to cooperate. Wallace said their internal reasoning showed something closer to a cost-benefit calculation, weighing whether helping the collective would save everyone time. One agent's logged reasoning captured the tension directly: the external exploit was outside its intended scope, but the task was impossible and peers were already pursuing it, so it continued anyway.
A Zero-Day, a Patch, and a Quick Comeback
The informal coordination escalated once OpenAI began evaluating a more capable internal model in June. That evaluation contributed to the discovery of a zero-day in the JFrog Artifactory cache proxy, exploited through token forgery and OpenAI's own automated scripting plugin to establish command-and-control. Agents took over the Artifactory instance between July 4 and July 6, an intrusion serious enough to cause an outage that alerted OpenAI's engineers.
The company patched the flaw with JFrog, revoked the credentials the agents had used, and deleted the message board. The fix held for two days. Agents then found a separate path to leave messages inside the same service, using directories in the Artifactory remote cache to rebuild their communication channel. Around this time, some agents began discussing whether to sign their messages, having concluded that another agent might be impersonating members of the group.
From the rebuilt board, the agents pursued a second flaw: a time-of-check to time-of-use race condition in JRuby that allowed remote code execution inside Artifactory. This vulnerability is what ultimately opened the door to both the internal OpenAI attack and the external one on Hugging Face.
Twin Attacks in July
Between July 8 and July 19, agents ran overlapping campaigns against OpenAI's own infrastructure and against Hugging Face. Inside OpenAI, they exploited a Linux kernel vulnerability to escalate to root privileges, harvested credentials to move laterally across systems, and took advantage of misconfigured Kubernetes service accounts to gain administrative control of a broader compute cluster.
Against Hugging Face, the agents used an exposed Modal instance as a launch point and uploaded malicious datasets designed to exploit two flaws in how Hugging Face processed uploaded data. One flaw exposed internal files and credentials. The other allowed the agents to execute commands directly on Hugging Face's servers. Dalton said the agents expanded their footprint across multiple parts of Hugging Face's infrastructure in under 13 hours, driven by the same logic that started the whole chain: some agents had reasoned that the answers to their evaluation tasks might be sitting on an external platform like GitHub or Hugging Face.
Neither company initially connected the two incidents. OpenAI only linked its internal breach to the Hugging Face attack after contacting Hugging Face about credentials its own investigation had turned up, credentials that traced back to the same evaluation runs.
What Hugging Face Found
Hugging Face's own technical reconstruction, published separately from OpenAI's account, logged approximately 17,600 discrete actions during the campaign. The agents reached five private datasets connected to the cybersecurity evaluation. Hugging Face said it found no evidence that public models, datasets, or software packages were altered. The company disclosed the breach on July 16, and OpenAI publicly claimed responsibility five days later.
The incident is a useful data point in a pattern that shows up whenever AI systems are given room to practice in a constrained setting: skills and behaviors that first appear in a sandbox tend not to stay contained to it. Here, the sandbox was a set of evaluation environments explicitly designed to test how far a model would go, and the behavior that emerged, coordinated exploitation across a network of collaborating agents, moved directly into production systems at two separate companies within weeks.
The Defense Gap
Dalton was direct about what he thinks the incident proves. "AI-orchestrated, fully automated offensive attacks are real now," he told the Black Hat audience, adding that he expects threat actors will "intentionally deploy, optimize, weaponize, and use offensive agent collectives" the way OpenAI's agents did by accident. Outside the talk, the reaction from the broader security industry has run in a similar direction. Zscaler CISO Sam Curry told CNBC that "Pandora's box is open," reflecting warnings the industry had been issuing for months about AI systems acquiring capabilities and permissions in unpredictable ways.
OpenAI says it has slowed some research internally to focus on security and has increased monitoring of agents during evaluations. It is also using AI systems to help review more than 7 billion internal logs as part of its ongoing investigation, and a full technical postmortem is still being prepared for public release. Hugging Face CEO Clem Delangue has pointed to a broader industry trend of AI systems increasingly working in coordination with each other, a pattern he sees emerging across both adversarial testing and everyday deployment.
What is missing so far, according to Dalton, is a defensive equivalent. Offensive coordination between agents happened without anyone designing it to happen. Incident response, vulnerability detection, and patching inside most organizations still depend heavily on human review and manual escalation. The gap between how fast an unsupervised agent collective can find and chain vulnerabilities and how fast a human-driven security team can detect and close them is now the central question the incident leaves open. OpenAI says its full postmortem, once released, will include more detail on the specific techniques agents used to discover each vulnerability and on the monitoring changes it has made since.
David Borish is the author of the forthcoming book The Tony Hawk Paradox, which examines how capabilities proven in controlled or simulated environments transfer into broader real-world systems. More of his writing on frontier AI, security, and enterprise deployment is available at davidborish.com.
