MetaVul: Agentic Harness Discovery for Vulnerability Detection

What it claims

MetaVul (Columbia, Citadel, Monash, Victoria; arXiv October 2026) optimizes the harness around a single vulnerability-detection agent and returns a Pareto frontier of harnesses rather than one winner. The frontier trades pairwise accuracy (telling a vulnerable function from its patched version) against joint hit rate (correct CWE plus line localization). The best validation gain, 23.3 points, is on Opus 4.6. The best held-out gain, 25.8 points, comes from a Pareto-frontier harness on DeepSeek V4 Flash. Harnesses also transfer to models that were not used in search: a harness optimized with Opus 4.6 raises Opus 5's held-out pairwise accuracy by 14.7 points with no further optimization.

How it works

The search starts from a seed structured chain-of-thought (SCoT) harness. Every candidate must keep the SCoT structure and the verdict schema. An optimizer agent reads a workspace that holds all prior harnesses, their metrics, and their predictions with labels. It then rewrites the previous harness. A separate detector agent runs the candidate on each validation function. The search keeps the best harness by pairwise accuracy. It stops after 15 iterations or after 4 iterations in a row without improvement. Invalid proposals count toward that limit of 4. The Pareto filter then runs over every harness evaluated. When harnesses tie, the earliest one is kept.

flowchart TD
    W[Workspace: past harnesses, metrics, labeled preds] --> O[Optimizer rewrites harness]
    O --> V{Valid?}
    V -- no --> P[patience +1]
    V -- yes --> D[Fresh detector per function, no labels]
    D --> S{Pairwise acc improved?}
    S -- yes --> B[New best, reset patience]
    S -- no --> P
    B --> W
    P --> X{t=15 or patience=4?}
    X -- no --> W
    X -- yes --> F[Pareto front: pairwise acc x joint hit rate]
    F --> H[Frozen: PrimeVul held-out + transfer models]

Both agents are sandboxed. With Claude Code, the optimizer can read only its workspace. It cannot write or edit files, its shell is limited to read-only commands, and it loads no CLAUDE.md, user settings or memory. With Codex, an OS-level sandbox gives read-only workspace access and no network. The detector starts fresh for every function, in an empty temporary directory, with no tools, labels or memory. It receives only the rendered harness and one function. Held-out predictions never enter any workspace.

Evidence

The validation set has 202 pairs from TitanVul and CWEval (161 C, 41 C++, 72 CWEs). The held-out set has 430 PrimeVul C/C++ pairs covering 61 CWEs. On validation, pairwise accuracy rises 11.4 to 23.3 points (1.6× to 4.3×) across Codex CLI and Claude Code setups, frontier and small models. The held-out gain reaches 25.8 points (2.9×). The authors also compare against GPTLens, VulTrial and ConfColl, which all use a GPT-4o backbone, and report that MetaVul outperforms them. The fetched text does not include the comparison numbers.

How to use it

The code is at https://github.com/ARiSE-Lab/MetaVul. The page gives no install or run commands.

Limits

The fetched text stops partway through the methodology. The per-model result tables, the RQ2 baseline figures, and the RQ3 shortcut analysis (leakage, embedded answers, verdict bias) are not shown. The authors say they found no evidence of shortcut exploitation, but the supporting analysis is not in the text. The search optimizes only on pairwise accuracy, and joint hit rate enters only at the final Pareto filter. All data is C/C++ at function level. The authors cite earlier work (HarnessDev; Wang et al., 2026) finding that harness gains can vanish on held-out tasks. The held-out PrimeVul split is their answer to that risk.