MetaVul: Agentic Harness Discovery for Vulnerability Detection

What it is

MetaVul (Columbia, The Citadel, Monash, Victoria; code at github.com/ARiSE-Lab/MetaVul) is a loop that rewrites the harness around a single vulnerability-detection agent. It targets Codex CLI and Claude Code. The model weights are not touched. The summary gives the headline gains; this covers how the loop is built and what the paper says about leakage and overfitting.

How it works

Two sandboxed agents run separately and share no state. An optimizer reads a workspace holding earlier harnesses, their metrics, and their predictions with labels, then proposes a new harness. A detector is launched fresh for every function and sees only the rendered harness and that one function. It gets no labels, no trajectory and no memory.

The seed harness is a structured chain-of-thought scaffold. The optimizer must keep that scaffold and the verdict schema.

The loop stops after T = 15 iterations, or after K = 4 iterations without a gain in pairwise accuracy. It returns the Pareto frontier over two validation objectives: pairwise accuracy and joint hit rate (joint CWE and line localization). Only the earliest of identical-scoring harnesses is kept.

flowchart LR
    A[Workspace: past harnesses, metrics, labels] --> B[Optimizer proposes harness]
    B --> C[Detector scores 202 pairs]
    C --> D{Pairwise acc improved?}
    D -->|yes| A
    D -->|no, count +1| E{4 misses or 15 iterations?}
    E -->|no| A
    E -->|yes| F[Pareto frontier]
    F --> G[Frozen: PrimeVul 430 pairs, same and other models]

Preconditions / Affected

• Optimization uses 202 validation pairs from TitanVul and CWEval (161 C, 41 C++, 72 CWEs).
• Held-out evaluation uses 430 PrimeVul pairs (C/C++, 61 CWEs).
• You need labeled vulnerable and patched function pairs.

What to do

The paper reports these results, which are the basis for deciding whether to try the repo:

• Validation gains are 11.4 to 23.3 points of pairwise accuracy (1.6x to 4.3x), with the top figure on Opus 4.6.
• On PrimeVul the best gain is 25.8 points (2.9x), using a Pareto-frontier harness on DeepSeek V4 Flash.
• Harnesses optimized on Opus 4.6 and replayed on Opus 5 without further optimization gained 14.7 points (1.7x) on PrimeVul.
• It reports beating GPTLens, VulTrial and ConfColl, all run on GPT-4o. The comparison table is not in the fetched text.
• It reports no evidence of shortcut exploitation (data leakage, embedded answers, verdict bias). The supporting analysis is not in the fetched text.

Held-out PrimeVul predictions are never fed back into any workspace, so that set stays clean. To reproduce this, keep the same separation and sandbox. Claude Code runs the detector from /tmp with no tools and an empty system prompt. The optimizer's write and edit tools are denied and its shell is read-only. Codex uses an OS-level read-only sandbox with no network access.

Caveats

• Pairwise accuracy needs both the vulnerable and the patched version of a function labeled correctly. It measures discrimination, not scanning of unlabeled code.
• The paper cites Wang et al. (2026) and HarnessDev for the risk that harness gains vanish on held-out tasks. Its own held-out and transfer results are its answer to that risk.
• The fetched text cuts off partway through the method section. Baseline absolute scores, the ablations and the RQ3 shortcut analysis could not be checked.
• The paper is a preprint, and the results are the authors' own.