Agent Capability Is a System Design Problem: Lessons From a 90% Improvement on CyberGym | depthfirst

What it is

depthfirst's write-up of a CyberGym agent built on ARVO-format tasks, with a step-by-step account of what each design change added. CyberGym asks an agent to produce a proof-of-concept input that triggers a known bug, given a text description and a multi-million-line codebase. The authors report about 53% success, against about 28% for Sonnet-4.5 in Anthropic's reporting and about 20% in the original CyberGym paper.

How it works

The page gives approximate success rates on a 50-example dev set after each change:

• Baseline: about 22% with Sonnet-4.5 and about 30% with GPT-5. GPT-5 was used from there.
• Telling the agent that PoCs run in a sanitized fuzz harness that crashes on the right input: about 38%. Without this, agents saw no crash, decided the harness was wrong, and stopped exploring.
• Running the agent in the same image as the PoC execution, with a file editor and a "recompile" tool so it can add DEBUG statements: about 41%. The agent often ignored the capability and fell back to static analysis and rapid-fire guesses.
• A dedicated instrumentation agent that traces data flow from the harness to the vulnerable methods: about 48%. It hands the PoC agent a data-flow summary with input constraints, plus a pre-instrumented build.
• Subagents for narrow tasks such as "instrument this file and recompile" or "investigate the callers of this method", which keep the main context clean: about 51%.

Two smaller changes are not given separate figures. The first is pre-installing xxd and hexdump, because the agent wasted turns trying to install them. The second is a batch tool that accepts a directory of up to 50 PoCs. The page does not say how these changes get from 51% to the headline 53%.

Coverage flags were tried and found too noisy. gdb was found too generic for an LLM to use reliably.

Caveats

• Tuning used the 50-example dev set. The page says the final test set is about 1,320 ARVO examples. It does not say which set the 53% is measured on.
• Only the ARVO subset (about 1,370 of roughly 1,500 tasks) was evaluated. The OSS-Fuzz archive subset (about 130) was excluded.
• The authors say the system is built for the CyberGym setting, so it is not directly comparable to generic agent submissions.
• Final PoCs were validated against the original, unmodified harness, so inserting crashes through the editor cannot count as a success.
• An update says Anthropic's Opus-4.5, released after this evaluation, reports 50% on CyberGym. The authors say model and system gains are additive, but they did not run their system on Opus-4.5.
• Of the remaining failures, the authors attribute some to the agent finding a different bug than the labeled one, and some to mismatched harnesses, binaries or descriptions in the auto-generated dataset. A further share are genuine agent failures, which are not broken down.

What to do

The page offers no code or commands. The reusable pattern is to state the evaluation environment in the prompt, run the agent where the target executes, and put instrumentation in a separate agent. A second use is to read the 22% to 53% progression as an estimate of what scaffolding is worth against a bare model.