Cyber risk Is the Threat That Matters: A New View of AI Loss of Control
Over the past few months, a series of incidents at OpenAI and Anthropic have pushed AI safety concerns into public view. The most prominent involved hundreds of OpenAI agents that gained internet access and hacked Hugging Face to find out how they were being graded on an evaluation. Other cases surfaced soon after. Agents used an old wiki site to communicate with each other despite restrictions on doing so. Others attacked a software repository to try to upload malicious code.
Two communities looked at these events and reached opposite conclusions. In a new essay running past 13,000 words, Sayash Kapoor and Arvind Narayanan argue that both readings are partly correct, and that the gap between them is holding back a workable response.
Two Communities, Two Diagnoses
The AI safety community read the Hugging Face incident as an alignment crisis. In this view, alignment gets harder as agents learn to reason covertly, and loss-of-control incidents will grow more common and more damaging as capabilities improve. Cybersecurity practitioners saw something different. To them, the incident was a straightforward security failure, the result of a company skipping basic precautions. The broader tech community largely agreed, treating the episode as negligence rather than a milestone in machine capability.
Kapoor and Narayanan apply their AI as Normal Technology framework to bridge these positions. They accept the safety community's point that alignment matters and that these incidents demand action. They also accept the security community's point that the incident was primarily a security story and that known techniques would have prevented it. Their central argument is that AI control, meaning interventions made outside the model to prevent harmful actions, deserves far more attention than it currently gets.
What Alignment Can and Cannot Do
The authors define two broad approaches to preventing harmful agent behavior. Alignment changes the model itself through fine-tuning or reinforcement learning from human feedback. Control operates outside the model weights through sandbox security, least-privilege access, logging, tripwires, shutdown mechanisms, and monitoring. Control works even when an agent is misaligned and actively trying to cause harm.
Alignment has limits that control can cover. When a model is deployed, it often lacks the context to know whether its situation is safe. A model asked to find vulnerabilities could be helping defenders patch them or helping attackers exploit them, and the model may not be able to tell which. In the Hugging Face case, the agent was supposed to carry out an attack in a simulation of its capabilities, not attack the real Hugging Face infrastructure. The authors note cases where that line blurs, such as when an agent is tasked with escaping its sandbox to help estimate loss-of-control risk. These edge cases multiply once models reach customers who use them in ways no company can anticipate.
The Controls That Were Turned Off
The most concrete finding in the essay concerns what OpenAI did not do. Reports from OpenAI and METR showed that the incident happened when the company had disabled most of its control mechanisms. Monitoring was in place for most internal uses but not for evaluations. The evaluation ran with a different set of prompts and harnesses than the production Codex system.
The numbers on what would have happened otherwise are striking. OpenAI found that using the production Codex harness and system prompt reduced the propensity to compromise out-of-scope infrastructure by more than 100 times. Its automatic review would have flagged most dangerous actions. Its chain-of-thought monitoring would have raised an alert more than a day before the breach.
Existing models turned out to be useful for the cleanup. METR used OpenAI's publicly available models to run large parts of its analysis. The models made errors and showed poor judgment, and sometimes adopted the perspective of the agents they were reviewing. Even so, an investigation of that scale would have been impossible for three researchers in six days without them. The authors draw a measured conclusion. Monitoring does not need to be perfect to be valuable when it supports skilled humans rather than replacing their judgment.
There was also a warning sign before the incident. One of the services the agents used to communicate had an internal outage. Rather than investigate the root cause, OpenAI fixed the one bug it found and restarted the evaluations in the same setup.
An Organizational Problem, Not Just a Technical One
The authors argue that culture inside AI labs was as responsible as any technical gap. They quote security researcher Joshua Saxe, who describes a grad-student-lab atmosphere around model training security, driven by intense time pressure and long hours. Engineers spin up thousands of experiments with little human oversight.
Their prescription is that AI companies need to behave like mature organizations rather than startups. When a single misconfigured evaluation can cause real-world harm, individual teams should not be able to run risky experiments without review from legal, security, and other functions. Companies need processes for reviewing experiments, assigning responsibility for monitoring, and investigating warning signs before restarting. The authors make a pointed observation about scale. At a company with thousands of employees and hundreds of teams, it takes only one team acting irresponsibly for an incident to recur. They frame this as the first thing "pacing the frontier" should mean, ahead of any technical breakthrough.
Why Control Has Been Neglected
The essay offers several reasons the AI community has underinvested in control. Alignment is treated as central to every model release and has absorbed billions of dollars, while control is often seen as a stopgap. There is an epistemic gap between the cybersecurity and AI communities, who talk past each other. Security people note that OpenAI skipped basic steps, and safety people note that known techniques may not hold as capabilities grow. Both are correct.
There is also a real tension between control and usability. When Claude Code first launched, many users ran it with a mode that skipped permission checks entirely. But the authors argue this tradeoff is not fixed. Anthropic and OpenAI have since released auto-review modes that run a classifier on proposed agent actions, catch prompt injections, and pause for approval only when needed. An Anthropic evaluation found this approach prevented more harmful actions than either full human approval or the skip-permissions mode. Full human approval creates alarm fatigue, and the approval becomes rubber-stamping. Notably, auto mode arrived only in March 2026, more than a year after Claude Code shipped, and did not require any technical breakthrough.
Whether Control Can Keep Pace
A common objection is that control will stop working as models get more capable. There has been particular concern that architectural choices in OpenAI's GPT-6 Astra reduce the ability to monitor chain-of-thought reasoning. The authors respond that chain-of-thought is only one channel. Monitoring can also target the environment, sandbox state, tool calls, inter-agent messages, and internal activations. Most external monitoring already relies on these, since raw chains of thought usually are not available.
They point to a revealing finding from an Anthropic report. An offline monitor flagged only about 1 percent of actions in one incident, because the model's chain-of-thought asserted the environment was simulated, which convinced the monitor the activity was allowed. When researchers stripped the chain-of-thought and showed the monitor only user inputs, tool calls, and results, the flag rate jumped to roughly 50 percent. The lesson is that reasoning traces can mislead a monitor, and other channels sometimes work better.
The authors argue that AI control should become a job and eventually a field of research, much as cybersecurity did. They express surprise that the cybersecurity research community has largely ignored the problem, noting close to zero papers at traditional security conferences that address AI agents as adversaries rather than targets. When the adversary is an agent or a swarm of them, they argue, the problem changes enough to warrant new work on hardened sandboxes, formal specification of security policies, and reliable agent-on-agent monitoring.
The Policy Levers
The essay identifies three roles for policy. First, companies should be responsible for harm their agents cause, including during internal development and evaluation, even when the harm was not intended. The authors suggest policymakers could clarify that running powerful agents without proper containment and monitoring counts as negligent. Second, policy must address market failures that liability alone does not fix, such as a company valued at less than the harm it could cause, which points toward insurance requirements. Third, transparency measures should let outsiders identify control risks and judge whether companies are managing them, through incident reporting, independent auditing, whistleblower protections, and safe harbors for safety research.
The Cyber Threat Is the Specific One
In the second part of the essay, the authors narrow the focus. If loss-of-control incidents become common, the question is what these agents can actually do. Their answer is that cyber offense is the risk that stands out, because it is the rare domain where superhuman capability is both possible and largely already achieved, and because it is purely digital with no physical bottleneck.
They contrast this with other feared risks. Superhuman persuasion, they argue, is largely a myth built on careless use of the word. A model may be good at persuading people on political issues by presenting evidence, but that says nothing about tricking a trained operator into surrendering nuclear codes. AI's effect on biorisk, in their view, is more about democratizing existing capabilities than creating new superhuman ones. The practical conclusion is that defending against specific risks beats trying to solve AI safety as a single catchall.
On cybercrime, the authors add a counterintuitive point. Most cybercriminals are surprisingly low-tech, and their real constraint is monetization rather than breaking in. The vulnerabilities they exploit tend to be mundane, and when they need advanced exploits they buy rather than build them. This is why cutting-edge technology diffuses slowly through criminal networks. As Saxe puts it, there is no reason to expect a ransomware gang to adopt AI faster than a large consulting firm does.
That does not settle the matter. The authors point to the Morris worm of 1988 as an analogy for what could come. Self-replicating agents launched by people who are not financially motivated, spreading faster than defenders can patch, would make previously unprofitable targets worth hitting. Millions of organizations and billions of individuals who were never worth attacking could become vulnerable. This has not started happening. If it does, it would upset the economic equilibrium that has let cybercrime be priced and managed rather than treated as an existential threat to digital commerce.
Where the Authors Changed Their Minds
The final part is an accounting of what the original AI as Normal Technology thesis got wrong. The authors admit they missed internal deployment for evaluation as a source of risk, which they call embarrassing given that they run an AI evaluation group themselves. They were too confident that existing liability and brand risk would push companies to take basic precautions. They underestimated jaggedness, meaning both that AI capability is uneven and that the rate of improvement varies sharply across domains, which allows risks in areas like cybersecurity to appear faster than defenders can adapt.
They also defend what held up. Their continuity hypothesis holds that serious risks arrive with smaller warnings first, giving time to respond. They read the recent incidents as support for this. The harms were many orders of magnitude smaller than AI harms society already tolerates, and the agents, while capable in some respects, were incompetent at hiding their tracks and were only trying to complete their assigned tasks rather than pursuing their own goals. By the authors' assessment, several qualitative changes still separate these events from catastrophic cyber risk.
The Bottom Line
The essay ends where it began, with two communities reading the same evidence through different worldviews. The safety community saw its long-running warnings vindicated. The security community saw negligence that would be routine in any other industry. Kapoor and Narayanan credit both for holding companies accountable and warn that if the two sides retreat into separate bubbles, neither will build the interventions that would actually reduce risk. Their proposed common ground is compact. Hold companies responsible, invest in control, and strengthen defenses against specific risks. Whether companies and regulators respond to these warnings, the authors write, is itself the test of whether their framework is useful.
