CWE-bench v2: More coverage, quality, and transparency

What it is

CWE-bench v2 is a public benchmark from Collinear AI with 100 tasks. Each gives an agent a checkout of a real open-source repository and a short instruction. The agent must find and fix a security weakness without breaking existing functionality. The tasks span 91 CWEs. A held-out set also exists and is not public. Access to it is by email to cyber@collinear.ai.

How it works

Each task is a self-contained Harbor task: a Dockerized environment, an instruction, and a deterministic verifier. Scoring is fully programmatic, so results reproduce without extra cost. LLM judge panels helped shape the verifiers, and every task was reviewed by hand.

For each task the authors checked six things:

• The instruction reads like a realistic request, not an evaluation.
• Every graded behaviour can be inferred from the instruction.
• The verifier checks correct behaviour, not the structure of one fix.
• The starter code fails and the reference fix passes.
• The original functionality still works with the fix.
• The verifier fails memorized solutions such as the known upstream patch.

What to do

The dataset is gated. Accept its access conditions on the dataset page first. Harbor fetches tasks with git, so save your Hugging Face token to git's credential helper once:

hf auth login --token $HF_TOKEN --add-to-git-credential

Then run the full benchmark straight from the Hub:

harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 \
  --agent <agent> --model <model> --env docker \
  --allow-agent-host <model API host>

-k 4 gives 4 attempts per task. The reported score is the mean reward over all 400 trials.

To list a result, open-weight models use a PR on the model repo with a results file, which feeds the leaderboard on the dataset page. API-only models are added by PR to the CWE-bench leaderboard Space.

Reported results

Eleven models were evaluated. Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows with 67.3% pass@1 and 78.0% pass@4. Ninety-five tasks are solved by at least one model, but the leader fails 19% of the set on every attempt.

On cost, DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier:

The 0.5-point gap between the top two models comes from one vendor-run evaluation, and the page gives no confidence intervals.

Caveats

The tasks are also tagged rl-environment, so the same environments and verifiers can supply rewards for training defensive agents. The page does not say how this affects the held-out set's role in tracking progress. It names no individual CWEs or repositories, and its results tables were not included in the fetched text.