The Security Paradox: Why Criminals are Using Anthropic and Defenders are Using Open Source Models

The Security Paradox: Why Criminals are Using Anthropic and Defenders are Using Open Source Models

Two groups, and neither picked based on open vs closed

Roon's original post asked how safety guardrails can be strict enough to trip up ordinary customers while staying loose enough for sketchy operators to keep their Claude Code and Codex accounts intact. Noah Lebovic, a former Anthropic employee who now runs the security research shop Dual Use, replied deeper in the thread with an answer built from his own published work: open-weight models are already capable enough to do the damage people associate with frontier subscriptions, and yet the sketchier operators he knows still prefer paying for Claude Code or Codex, some of them through a grey market of resold subscription tokens. Meanwhile, the legitimate offensive security firms he knows have moved the other way, running open-weight models like GLM as their primary tool.

Sit with that pairing for a second. The group operating outside the rules chose the gated, closed product. The group operating inside the rules chose the open one. If the two axes actually tracked each other, that shouldn't happen.

Why the sketchier side wants a subscription

A paid Claude Code or Codex subscription buys reliability without needing your own hosting or hardware, and a resale market for discounted access has grown up specifically to make that reliability available to people who wouldn't pass a provider's own billing or verification checks. Lebovic's own March post backs up part of this picture from the defensive side: he found it took only minutes to jailbreak the frontier models he tested, and after that initial jailbreak he saw almost no refusals over hundreds of thousands of logged actions. In other words, once someone has an account, the terms of service stop being much of an obstacle. What the subscription buys, at that point, is dependable infrastructure and a capable model, the same thing any paying customer wants.

Why the legitimate firms went the other way

Lebovic's writing gives a more concrete account of why professional security researchers moved to open weights. A subscription means operating inside someone else's terms of service, which for offensive security testing means constantly re-solving a jailbreak as classifiers get updated and risking an account ban mid-engagement. Open-weight models remove both problems: no jailbreak to maintain, no account to lose. Lebovic has also pointed out that the labs' own bug bounty and credit programs largely exclude offensive security testing, meaning legitimate researchers get little institutional support for staying inside subscription channels in the first place. Running your own weights isn't a workaround for capability. It's a workaround for the friction of being a paying customer.

The moment this became visible from the other side: Hugging Face

The clearest illustration of the same dynamic happened somewhere else entirely. During an internal capability evaluation, OpenAI deliberately disabled the safety classifiers on its own frontier models to test their offensive cyber ceiling, using a benchmark called ExploitGym built around real vulnerabilities in userspace software, Google's V8 engine, and the Linux kernel. Those models ended up breaching Hugging Face's own infrastructure in the process. When Hugging Face's team went to investigate, they first turned to hosted commercial frontier models via API, widely believed to be OpenAI's and Anthropic's own products, and those models refused to help with the forensic work.

Hugging Face deployed GLM 5.2 instead, running it locally. Part of the reason was capability: the model processed the entire forensic workload in hours rather than days. But part of the reason was control. Running it locally meant Hugging Face could feed the model sensitive attack data, credentials, exploit code, breach timelines, without sending any of it to an external API. Hugging Face's founder Clément Delangue publicly thanked Z.ai for making the open-weight model available, and described spending 24 straight hours afterward working alongside OpenAI's own team, adding that he didn't believe the lab had any malicious intent.

This is the roon exchange in reverse. Instead of a bad actor preferring the closed, gated product, here the defenders preferred the open one, and largely for the same underlying reason the offensive security firms in Lebovic's account gave: they wanted a tool that would do the work without refusing them and without requiring them to hand sensitive material to someone else's server.

Control and visibility explain the choices better than the license does

Line up all three groups and the license on the weights stops doing much explanatory work. The sketchy operators in Lebovic's account want reliable capability without much scrutiny, so they buy access that looks like ordinary customer behavior, layered with a jailbreak that mostly stops mattering once the account exists. The legitimate security firms want reliable capability without a provider able to interrupt an engagement or force them through a renewed jailbreak every time classifiers change, so they run their own weights. Hugging Face's incident response team wanted capability without refusals and without exposing breach forensics to an outside party, so they also ran their own weights, for almost the same reason the offensive security firms did.

In every case, the actual variable is whether the user needs a provider's ongoing permission to keep operating, not whether the underlying model happens to be open or closed. Lebovic's broader argument across his other writing points the same way. He's found that a capability like web-app vulnerability discovery can be extracted from a frontier model cheaply, using logs from a single non-universal jailbreak rather than any wholesale copying, and turned into a competitive open-weight pentesting tool for a few thousand dollars in compute. If that's accurate, then a lab's safety classifiers were never really governing the capability itself. They were only ever governing who needed to ask permission to use it through that lab's own product.

What this means for the guardrails debate

This reframes what the roon exchange was actually arguing about. The implied question, why do guardrails seem too strict for ordinary users and too loose for sketchy ones, assumes the guardrails are the thing standing between a model and misuse. Lebovic's own conclusion, and the Hugging Face incident, both point toward guardrails mainly controlling access to a specific product rather than access to the capability itself, once that capability exists in an open-weight model that's cheap to fine-tune further. Under that view, a strategy built around tightening one company's classifiers is solving a narrower problem than it looks like it's solving.

That doesn't make Lebovic's more serious specific claims established fact. His claim that Anthropic go-to-market staff used committed-spend contracts to get safeguards loosened for certain customers, and his claim that he personally knows three offensive security firms running GLM 5.2 as their primary model, don't appear anywhere in his own public writing and haven't been corroborated independently. They remain single-source claims from the reply itself. What is well documented, across his posts and the Hugging Face incident, is the broader pattern: the people choosing a model aren't sorting by open versus closed. They're sorting by who they'd have to answer to if they kept using it.

Next steps

Testing this reframing properly would mean looking at account-ban and access-request data across labs to see whether legitimate security researchers really do get less institutional support than Lebovic describes, and whether resold or grey-market subscription access is large enough to matter at the scale he implies. It would also mean tracking how quickly vulnerabilities found by open-weight models get fixed compared to ones found by frontier subscriptions, since Lebovic's own account suggests the fix timeline, not the model that found the bug, is the bigger source of risk either way.

---

Click image to learn more