arxiv.orgpaperAI Security Briefing
This arXiv paper introduces a cost-aware evaluation framework for LLM security agents, comparing models at fixed inference and tool-spend budgets rather than only reporting best-case success rates. It tests offensive agents on Cybench CTF challenges and defensive agents on Splunk BOTS v1 SOC investigation challenges, finding that offensive CTF performance scales well with added test-time compute (open-weight models can approach frontier proprietary ones cost-competitively) while defensive SOC investigation success depends more on disciplined tool use and telemetry navigation than raw reasoning budget.
TL;DR A cost-aware benchmark shows offensive LLM agents scale with compute on Cybench CTFs, but defensive SOC agents on Splunk BOTS v1 depend more on disciplined tool use than inference budget.
Techniques named here
- Vulnerability discovery
- Exploit development
- Penetration testing
- CTF completion
- SOC investigation / telemetry navigation
Related in AI Security
blog.talosintelligence.comtoolIntroducing CAIRN: Frontier tracking for AI-integrated malware (Cisco Talos)
github.comadvisoryFlowise MCP Security Bypass Enables RCE
github.comrepoYesWeHack Claude Kit
jfrog.comarticle