CWE-bench v2: More coverage, quality, and transparency

What it claims

CWE-bench v2 from Collinear AI is a security-repair benchmark. An agent gets a checkout of a real open-source repository and a short instruction, and must find and fix a weakness without breaking existing functionality. The public set has 100 tasks covering 91 CWEs. A separate held-out set tracks progress on unseen tasks, and models are evaluated on it by contacting cyber@collinear.ai.

How it works

Scoring is fully programmatic and deterministic. LLM judge panels helped curate the checks toward hardness and fairness, and every task was also reviewed by hand. Each task was checked against six criteria:

• The instruction reads like a realistic request, not an evaluation.
• Every graded behaviour can be inferred from the instruction.
• The verifier tests correct behaviour, not the structure of one fix.
• The starter code fails and the reference fix passes.
• Original functionality still works after the fix.
• Memorized solutions, such as the known upstream patch, fail the verifier.

Evidence

Eleven models were evaluated. Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows at 67.3% pass@1 and 78.0% pass@4. Of the 100 tasks, 95 are solved by at least one model. Gemini 4 Argon fails 19% of the set on every attempt.

On cost, three models form the Pareto frontier. DeepSeek V4.1 Flash (1017) reaches 57.0% pass@1 at $0.04 per rollout. Claude Opus 5.5 adds about ten points at $1.83 per rollout. Gemini 4 Argon edges ahead at $3.92 per rollout.

How to use it

Each task is a self-contained Harbor task: a Dockerized environment, an instruction and a deterministic verifier. The dataset ships a registry.json, so Harbor runs the benchmark directly from the Hugging Face repo:

harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 \
  --agent <agent> --model <model> --env docker \
  --allow-agent-host <model API host>

-k 4 runs 4 attempts per task. The reported score is the mean reward over all 400 trials. The dataset is gated, so its access conditions must be accepted on the dataset page first. Harbor fetches tasks with git, so the HF token must be saved to git's credential helper once:

hf auth login --token $HF_TOKEN --add-to-git-credential

Results appear in two places. The leaderboard on the dataset page covers open-weight models and is updated by opening a PR with a results file on the model's repo. The CWE-bench leaderboard Space covers all models, including API-only ones, and new entries are added by a PR on the Space. The tasks are tagged rl-environment, and the page states that the same environments and verifiers can supply rewards for training defensive agents.

Limits

The public set is 100 tasks, so each task moves pass@1 by about one point. Five tasks were solved by no model, and the page does not say whether they were re-verified as solvable beyond the reference-fix check. Held-out results require contacting Collinear, and the page reports no held-out numbers. The page does not state a licence for the dataset beyond its gated access conditions.