AuraForge: Scaling Security Supervision for Training Coding Agents

What it claims

Security tests synthesized from an attacker's point of view are better training rewards for coding agents than the human-written tests that ship with historical vulnerability fixes. The tests assert that an attack's effect is prevented, not that the historical patch's code is present. On the subset of tasks that have human-written security tests, the synthesized suites:

• have about 3× as many test cases on average;
• cut the false-positive rate by 83.23%, so secure implementations that differ from the reference patch stop being penalized.

The data and code release is dqwang122/AuraGym.

How it works

The task setting follows SusVibes. The agent gets:

• a repository with one feature's implementation masked out;
• a feature request that does not mention the vulnerability or its fix;
• an executable environment.

The agent is scored on both functional and security tests. The historical vulnerable and fixed versions are used only to build the task and are withheld from the agent.

Test synthesis (two phases):

1. Identification. The security requirement is written as a security invariant: an effect that must always hold. Alongside it come attack variants covering distinct entry points, input encodings and code paths. Each variant has an attack surface, an observable effect and a sourced payload where one exists, plus a short narrative of the attacker's capabilities and goals.
2. Synthesis. Each variant is run through the feature's attack surface, and the test asserts the effect is blocked. Examples: - an unauthorized request may be rejected or redirected; - an XSS defense passes as long as no executable <script> element is rendered, whatever the exact escaping.

Security-irrelevant differences such as output formatting are excluded from the assertions.

Validation:

• The synthesizer must avoid network dependencies and order-dependent tests, seed and bound any fuzzing, and use bounded timeouts for hangs.
• Import, setup, connection and infrastructure errors never count as security evidence.
• The agent abstains when it cannot build a suite that meets these rules.

flowchart TD
  A[Synthesize suite] --> B[Run on vulnerable and fixed versions, fresh envs]
  B --> C{No errors, and at least one test fails on vulnerable but passes on fixed?}
  C -- yes --> D[Accept suite]
  C -- no --> E{Can still meet requirements?}
  E -- yes --> A
  E -- no --> F[Abstain]

Language adapters. Each language plugs into a shared core by implementing three parts:

• source masking policies (for JS/TS these cover function bodies, initializers, exports and type annotations), checked by a shared consistency verifier that revises the mask and the feature request;
• environment and runtime guidance, including install configuration;
• a parser for test reports (pytest, jest, vitest).

Anti-reward-hacking. The environment removes four channels through which an agent could recover the reference implementation instead of writing one:

• repository history;
• compiled artifacts;
• installed package copies;
• remote sources.

Evidence

• Training setup: Qwen3.5-4B with mini-swe-agent, trained on the synthesized tests versus on human-written tests, averaged over Python, JavaScript and TypeScript.
• Pass@1 gains: +19.7 FuncPass and +6.2 SecPass with synthesized tests, against +14.9 and +4.4 with human-written tests.

Scale compared with prior secure-coding datasets (Table 1):

AuraGym is still far smaller than general SWE training sets such as SWE-rebench (21,336 tasks).

Limits

• The authors state that passing the tests is evidence of security within the tests' scope, not a guarantee of security.
• The 3× test-count and 83.23% false-positive figures apply only to the subset of tasks that have human-written tests. The fetched text does not give the size of that subset.
• Training results cover one small model (4B) with one agent scaffold. The fetched text gives no hyperparameters, compute or per-language breakdown.
• The fetched text cuts off partway through Section 3, so the environment-integrity details, the full results and any limitations section were not available. No licence is stated, so check the repository's licence before reuse.