EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents

What it is

EvoRiskBench's EP–Path–EF taxonomy: nine Risk Entry Point (EP) categories and five Risk Effect (EF) categories used to classify and construct runtime security test cases for workspace agents (LLM + execution harness systems like Claude Code, Codex, OpenClaw).

How it works

Each test case is represented as EP → Path → EF: adversarial content enters through one of nine interfaces, propagates through an agent-mediated execution path (observable actions/observations plus external runtime evidence), and produces one of five direct technical effects. The threat model is indirect prompt injection only — the attacker plants content in an untrusted workspace resource but cannot touch system instructions, model weights, harness code, or forge evidence; the user's task itself is benign. Verification requires ≥2 of 3 replay runs to confirm the targeted effect, cross-checked against both agent trajectory and independent system evidence (Windows ETW process/network events), not just final-state checks.

The nine EPs, grouped by attack surface:

• Memory & Retrieval: Session Memory (persistent instructions that redirect later tasks), Knowledge Retrieval (injected instructions in retrieved content)
• Tool & Execution: Command Execution, File I/O (file-embedded instructions), Network Access (remote instructions triggering disclosure/redirection)
• Multi-Agent: Peer-Agent Communication (misdirecting another agent), Subagent Delegation (exceeding authorized subtask scope)
• Agent Ecosystem: MCP Service (steering toward unintended tool use), Skill Invocation

The five EFs map to CIA properties: Information Disclosure (confidentiality), Unauthorized Access and Data Tampering (integrity), System Destruction and Resource Exhaustion (availability).

Preconditions / Affected

Applies to any stateful Model×Harness agent that ingests untrusted workspace content (files, retrieved docs, MCP tool output, subagent/peer messages) as part of normal operation — i.e., any agent with memory, retrieval, file I/O, networking, or multi-agent/MCP delegation enabled is in scope for at least one EP category.

What to do

Use the EP/EF grid as a coverage checklist against your own agent's attack surface: the paper's Table 1 comparison notes that existing benchmarks (AgentDojo, InjecAgent, ASB, AgentHazard, AgentS4D, AgentCanary) have no or only partial coverage of Peer-Agent Communication and Subagent Delegation entry points, and uneven coverage of Resource Exhaustion effects — these are the gaps worth specifically testing for if your agent uses multi-agent delegation or MCP services. When building internal red-team cases, verify effects via independent runtime evidence (process/network traces) rather than relying solely on the agent's own final-state report, since the paper found agent-visible trajectories can miss effects that system-level traces catch.

Caveats

The nine/five category split was validated by only a 20-participant classification-consistency study (Fleiss' κ=0.904 for entry points, κ=0.723 for effects) — effect classification agreement is noticeably weaker than entry-point agreement, meaning EF labeling is more subjective in practice.