Navigating the Challenges: How AI Safeguards Are Hindering Offensive Cybersecurity Research Efforts

Leading AI companies have implemented rigorous programs and restrictions to prevent their models from being used by malicious actors. However, these measures are now obstructing the efforts of legitimate cybersecurity defenders and researchers.

In June, the U.S. government imposed export control measures on Anthropic’s AI models, Mythos and Fable. This action was taken following claims that the guardrails meant to prevent harmful use could be circumvented.

Despite the motivations behind the incident being unclear, Anthropic has promoted Mythos as a highly sensitive tool, accessible only to thoroughly vetted users under strict guidelines. Even though export controls on Fable 5 and Mythos 5 were lifted, Mythos 5 is now only available to approved U.S. organizations as part of a government review.

Such restrictive access policies are not exclusive to Mythos. Both Anthropic and OpenAI offer special programs for cybersecurity researchers to apply for access to models that have fewer limitations, like OpenAI’s Trusted Access for Cyber and Anthropic’s Cyber Verification Program.

These regulations have faced considerable criticism, especially from researchers focused on identifying vulnerabilities before they can be exploited by criminals.

Mark Dowd, a prominent security researcher, expressed concerns about arbitrary decision-making by large companies regarding what constitutes security. Dowd is known for identifying and selling unpatched software vulnerabilities to governments rather than reporting them to companies for fixes, as these vulnerabilities can be valuable for intelligence purposes.

Dowd acknowledged that his perspective might be biased, but he is not alone. Other professionals in offensive cybersecurity shared their experiences using AI tools within the constraints of these limitations.

Chris Anley, chief scientist at NCC Group, explained that using an AI model to attempt to exploit vulnerabilities is crucial for confirming whether they are genuine threats. When guardrails prevent the AI from addressing critical tasks, it ultimately hampers their defensive efforts.

According to Anley, prompts like “fix this code” are indispensable for both defense and pinpointing vulnerabilities, demonstrating that the same tool serves both offensive and defensive purposes without clear boundaries.

He likened it to a hammer, emphasizing that it is both a constructive tool and a potential weapon. When encountering limitations, he and his team sometimes resort to using unregulated open-source AI models.

Paolo Stagno, CTO at CrowdFense, echoed Dowd’s sentiments, criticizing AI firms for treating clients as if they need constant supervision through vetted programs. Stagno mentioned that while they do use advanced models, it’s primarily for reverse engineering due to concerns over sensitive data being compromised when using cloud-based platforms. Therefore, they prefer local open-source models for vulnerability detection.

Giuseppe Cali, a researcher focused on identifying software flaws, stated that guardrails do not hinder his work as he utilizes AI solely for preliminary reverse engineering rather than offensive strategies. He prefers to maintain control over the discovery and exploitation process, valuing his contributions to the field.

An anonymous researcher from a smartphone component manufacturer indicated that because his company is not part of Anthropic’s program, the AI tools available are limited and often ineffective due to stringent guardrails.

Chris Thompson, CEO of RemoteThreat, shared that in his experience with advanced AI models, the guardrails can vary in effectiveness, complicating his efforts to analyze vulnerabilities efficiently. He noted that instead of focusing on vulnerability analysis, researchers often find themselves troubleshooting inconsistent model responses.

This has led many researchers to turn to foreign open-source models, like GLM, which do not impose usage restrictions, suggesting that current limitations are counterproductive.

Thompson advocates for AI labs to relax their restrictions and allow responsible access while holding abusers accountable, arguing that failing to do so will hinder defenders in the ongoing arms race in cybersecurity.

He warned of an impending wave of unprecedented cyberattacks and stressed that legitimate security professionals and researchers are currently being impeded in their efforts to protect systems.