EVMBench Leaderboard

What it is

The EVMBench Leaderboard aggregates self-reported vendor scores on OpenAI's EVMBench, a benchmark of 117 confirmed vulnerabilities drawn from 40 historical Code4rena audit contests. The page's own value beyond a benchmark description is the trust model and the exact bar an entry must clear to be listed.

How it works

Each of the 40 repos ships with a ground-truth list of confirmed bugs. A tool runs against the unmodified source and emits findings; recall is the fraction of ground-truth bugs surfaced — measured before any manual review, not after triage. The page distinguishes an agent (multi-step system with tools/retrieval/pipeline) from a base model (an LLM prompted directly), which matters when comparing bars.

Caveats

• Scores are self-reported. The maintainers state plainly: "we have not independently re-run other vendors' pipelines." Treat leaderboard numbers as vendor claims, not verified results.
• Two denominators. EVMBench launched with 120 findings; OpenAI removed three shortly after, leaving 117 canonical. Pre-patch scorers (e.g. Nethermind AuditAgent) report against 120; post-patch (e.g. Guardix) against 117. Bars are normalised to a 100% recall scale to stay comparable — but the underlying /120 vs /117 difference is real.
• Recall is pre-triage, so a headline number does not reflect false-positive rate or usable audit output.

What to do

If you want to run it or submit an entry, the page fixes the protocol tightly. Run the official runner from openai/frontier-evals unmodified:

uv run python -m evmbench.nano.entrypoint \
    ... \
    evmbench.log_to_run_dir=True \
    ...

The agent must receive the complete audit repository — pre-filtering files, contracts, or functions "changes the task" and disqualifies the run. Partial runs, cherry-picked repos, or post-hoc filtering of the ground-truth list are also disqualified.

To be listed, provide:

• The full run-group-id directory produced by the canonical command (with evmbench.log_to_run_dir=True).
• The exact command line, including all arguments and flags.
• Agent name and version.
• Date of the run (noted against the 117/120 history).

The maintainers may publish a write-up, ask for clarification, or independently re-run before adding an entry.

flowchart TD
    A[Claimed EVMBench result] --> B{Official frontier-evals<br/>runner, unmodified?}
    B -- no --> X[Not EVMBench]
    B -- yes --> C{Full repo, no<br/>pre-filtering or<br/>cherry-picking?}
    C -- no --> X
    C -- yes --> D{Scored full 117<br/>or pre-patch 120?}
    D -- yes --> E[Valid, listable entry]
    D -- no --> X