Agent Capability Is a System Design Problem: Lessons From a 90% Improvement on CyberGym | depthfirst

What it claims

depthfirst reports about 53% success on CyberGym vulnerability reproduction, against about 28% that Anthropic reported for Sonnet-4.5. depthfirst argues that agent scaffolding, not only the base model, limits capability. The step-by-step figures in the existing summary (22% to 51%) come from a 50-example development set. The roughly 53% headline comes from a final test set of about 1,320 examples. Both sets are drawn only from the ARVO portion of CyberGym (about 1,370 of about 1,500 tasks). The 130 OSS-Fuzz-archive tasks were excluded because their runtime interfaces are less consistent. A later update notes that Opus-4.5, released after the evaluation, reports 50% on its own. depthfirst says model gains and system gains add together.

How it works

The prompt fix targets a specific failure mode. Without context about the harness, agents wrote plausible PoCs, saw no crash, decided the harness was broken, and stopped exploring. Telling the agent that PoCs run inside a sanitized fuzz harness, which will crash on the correct input, kept it iterating.

For runtime visibility, compile-time coverage flags were too noisy. gdb was too generic and unwieldy for the LLM to use reliably. What worked was running the agent in the same image where PoCs execute, with a file editor and a "recompile" tool, so it could insert DEBUG statements. On its own this gave only 41%, because the agent mostly ignored it in favour of static analysis and rapid PoC guesses. A dedicated instrumentation agent fixed this.

flowchart LR
  D[Bug description] --> I[Instrumentation agent]
  I --> S[Data-flow summary + input constraints]
  I --> P[Pre-instrumented build]
  S --> A[PoC agent]
  P --> A
  A -->|spawn| Sub[Subagents: instrument file, trace callers]
  Sub --> A
  A --> B[Batch test up to 50 PoCs]
  B -->|debug output| A
  A -->|edit + recompile| P

Subagents get narrow tasks such as "instrument this file and recompile". This keeps their exploratory output out of the main agent's context window. Smaller additions included preinstalled xxd and hexdump, so the agent does not waste turns on package installs. A batch tool also replaced CyberGym's single-file submission and accepts a directory of up to 50 PoCs.

Evidence

All step gains are on the 50-task dev set: baseline 22% (Sonnet-4.5) or 30% (GPT-5), harness context 38%, editable runtime 41%, instrumentation agent 48%, subagents 51%. Final evaluation validated every PoC against the original, unmodified harness, so inserted crashes could not count. The page gives no per-step numbers on the full test set, no cost or token figures, and no variance across runs.

Limits

depthfirst states that the system is built specifically for CyberGym and is not one-to-one comparable with generic agent submissions. The 28% baseline used a different harness and the full benchmark, not the ARVO subset. Of the roughly 47% of tasks still failing, some involve the agent finding a different bug from the labelled one. Others come from mismatched harnesses, binaries and descriptions in the automatically generated dataset. Some are genuine agent failures that the authors have not yet broken down. No code, prompts or system release is mentioned.