AuraForge: Scaling Security Supervision for Training Coding Agents

What it is

AuraForge is a pipeline (from CMU/UCLA researchers) for synthesizing executable security test suites from historical vulnerability fixes in real repositories, used to build AuraGym, a 679-task training/eval set spanning 344 repos, 177 CWEs, and Python/JS/TS. Code and gym are released at dqwang122/AuraGym.

How it works

Three stages, named in the paper as The Forge, The Aura, and The Seal:

• The Forge masks the historical security fix out of a repo while keeping the feature's interface intact, and writes a feature request that does not leak the vulnerability or its fix.
• The Aura has a synthesis agent write security tests from an attacker's perspective: it defines a security invariant plus several attack variants (different entry points/encodings/paths), then asserts on the attack's effect rather than the mechanism of the historical patch. This is explicitly to avoid false positives — e.g., it accepts either redirection or rejection as valid defenses against unauthorized access, rather than requiring the exact fix location.
• The Seal strips reward-hacking channels: repository history, compiled artifacts, installed package copies, and remote sources that could let an agent retrieve the reference fix instead of implementing one.

Each synthesized test suite is validated by running it against both the known-vulnerable and known-fixed implementations in a clean environment; it's only accepted if at least one test fails on the vulnerable version and passes on the fixed version, with infrastructure errors (timeouts, missing deps) never counted as security evidence.

Preconditions / Affected

Applies to teams building or evaluating RL/SFT training pipelines for coding agents where "security-aware" reward signal matters — not a tool you point at your own codebase today. Language support (Python/JS/TS) is adapter-based: adding a language means implementing three adapter components (source masking policy, environment/runtime config, test-report parser) without touching the core pipeline.

What to do

No commands or install steps are given in the fetched text — this is a research artifact, not a deployable tool. The actionable figures, if you're evaluating whether to adopt it for agent training:

• On the subset with existing human-written security tests, AuraForge generates ~3× as many test cases and cuts false-positive rate by 83.23%.
• Training Qwen3.5-4B with AuraForge-synthesized tests: +19.7 FuncPass / +6.2 SecPass, vs. +14.9 FuncPass / +4.4 SecPass with human-written tests.

Caveats

Results are reported for one model (Qwen3.5-4B) and one agent scaffold (mini-swe-agent); the paper does not claim generality beyond that setup. The CWE-coverage and repo counts (679 tasks / 344 repos / 177 CWEs) are from the released AuraGym construction, not an independent audit.