techcrunch.comarticleHow AI guardrails are impeding the work of offensive cybersecurity researchers | TechCrunch
TechCrunch reporting on how AI vendor guardrails and vetting programs (OpenAI's Trusted Access for Cyber, Anthropic's Cyber Verification Program) are frustrating legitimate offensive-security researchers, following export-control restrictions briefly placed on Anthropic's Mythos and Fable 5 models over jailbreak concerns. Named researchers from NCC Group, Crowdfense, and RemoteThreat describe inconsistent refusals when asking models to help confirm or exploit vulnerabilities, data-leakage worries pushing exploit work to locally-run open-source models, and a resulting drift of security work toward unrestricted Chinese models like GLM.
TL;DR Offensive security researchers describe how inconsistent AI guardrails on Claude/GPT models are hampering vulnerability research, pushing some to local open-source models like GLM instead.
Organisations named here
- NCC Group
- Chris Anley
- Chris Thompson
- Crowdfense
- Giuseppe Cali
- Mark Dowd
- Offensive AI Con
- Paolo Stagno
- RemoteThreat
Technologies named here
- OpenAI's Trusted Access for Cyber program
- Mythos
- Anthropic's Cyber Verification Program
- Fable
- GLM
- Open source AI models
- Open source models run locally
- Frontier AI models
- Anthropic
- OpenAI
- Claude Fable 5
- Fable 5
Techniques named here
- Jailbreaking AI model guardrails
- AI-assisted vulnerability confirmation
- Zero-day discovery and exploit development
- AI-assisted reverse engineering
Related in AI Security
blog.talosintelligence.comtoolIntroducing CAIRN: Frontier tracking for AI-integrated malware (Cisco Talos)
github.comadvisoryFlowise MCP Security Bypass Enables RCE
github.comrepoYesWeHack Claude Kit
jfrog.comarticle