AI Guardrails Are Pushing Legitimate Cybersecurity Researchers to Foreign Models
AI safety guardrails designed to block malicious cyberattacks are instead hindering legitimate defensive security research, pushing some researchers toward unvetted Chinese open-source models instead of U.S.-governed systems.
Anthropic and OpenAI have erected gatekeeping systems around their most capable AI models to prevent malicious use in cyberattacks. But the same safeguards are now blocking the work of offensive security researchers — the professionals tasked with finding vulnerabilities before criminals do.
The tension flared in June when the U.S. government imposed export controls on Anthropic's Mythos and Fable models after reports that their guardrails could be bypassed. Though those restrictions were later lifted, the underlying problem persists: both AI labs maintain vetted access programs (OpenAI's Trusted Access for Cyber and Anthropic's Cyber Verification Program) that require researchers to apply, prove legitimacy, and operate within strict usage boundaries.
Where Guardrails Block Legitimate Work
Chris Anley, chief scientist at NCC Group, a major security consulting firm, outlined the core dilemma. Asking an AI model to exploit a bug is essential for confirming whether a vulnerability is real and fixable. But if guardrails reject the request outright, the model becomes useless for defense.
"'Fix this code' as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities," Anley explained. The same query that helps patch software can also help attackers weaponize flaws — making the tool simultaneously defensive and offensive.
Paolo Stagno, chief technology officer at CrowdFense (a firm that acquires and sells unknown vulnerabilities to government agencies), said AI companies "essentially treat customers like children who need babysitting" through their vetted programs. His team avoids cloud-based models entirely for vulnerability work, relying instead on open-source models run locally to prevent sensitive data leakage.
One unnamed researcher at a smartphone-component manufacturer reported that his employer is not part of Anthropic's vetted program. As a result, the AI tools his team uses are nearly unusable for vulnerability discovery — the guardrails block any security-related queries outright.
Chris Thompson, CEO of RemoteThreat and founder of Offensive AI Con, observed that guardrails can be inconsistent even within vetted programs. Models enforce restrictions differently each day, forcing researchers to "spend a lot of time negotiating with the model instead of working on the core security program."
The Pivot to Unvetted Alternatives
These friction points are driving researchers toward Chinese open-source models like GLM, which can be downloaded freely and run locally without vetting or restrictions. According to Thompson, this trend is counterintuitive to national security interests: "You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems."
Not all researchers face the same barriers. Giuseppe Cali, a zero-day researcher, said guardrails have not impeded his work because he uses AI only for initial reverse engineering and code comprehension, not for finding vulnerabilities or building exploits. He prefers to own the vulnerability discovery process himself, a choice he said would remain unchanged even if all guardrails were lifted.
Mark Dowd, a security researcher known for selling zero-day exploits to Western governments rather than patching them, expressed discomfort with private AI companies making unilateral decisions about what is safe in security research. "It's not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what's not," he said.
The Trade-off
Thompson argued that the current approach causes more harm than good. Rather than tightening restrictions further, he called for AI labs to broaden access to vetted researchers and enforce accountability against those who abuse the tools.
Without such changes, Thompson warned, defensive security capabilities will lag behind the offensive threat landscape. "There's this big wave of attacks that are going to happen at speed and scale like never before," he said. "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now."
