SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

What it claims

Reasoning models usually write the code they will deliver inside their chain of thought, well before the final answer. Their own self-review rarely catches the security bugs in that draft. SecReason runs static security rules over the draft while it streams and injects warnings into the trace, so the model fixes the draft before delivery. The headline 72.8%–98.1% "secure and functionally correct" figure, up from below 15%, comes from a narrower setting than the summary implies:

• It counts only tasks where unassisted generation was vulnerable.
• It covers two locally deployed self-repair models.
• Streaming detection was evaluated on six models.

How it works

SecReason sits at the inference serving layer, between the inference engine and downstream clients. It does not change model weights. It watches code fences (e.g. ``` `py ```) inside the reasoning stream and parses tokens incrementally to detect completed statements.

Scanning on completed statements is a deliberate middle ground. Scanning at arbitrary token boundaries gives fragments that AST parsers reject. Waiting for the fence to close lets later statements build on the flawed one and leaves less of the trace for the model to react.

The two scanning tracks trade latency against coverage:

• Statement-local: checks each new statement against the ruleset as soon as it completes.
• Context-dependent: scans the accumulated code for cross-statement dataflows such as command or SQL injection. Scanning each statement in isolation would miss these, and rescanning everything on every statement would stall the stream.

When a rule fires, SecReason converts the structured finding (rule ID, line) into natural-language guidance and inserts it into the stream. Decoding then resumes from the existing token prefix. A new conversational turn would throw away the in-flight trace and force full regeneration.

flowchart TD
  A[Reasoning stream] --> B{Statement complete<br/>in code fence?}
  B -- no --> A
  B -- yes --> C[Statement-local check]
  B -- yes --> D[Dataflow scan of<br/>accumulated code]
  C --> E{Rule fires?}
  D --> E
  E -- no --> A
  E -- yes --> F[Interrupt, insert warning,<br/>resume from prefix]
  F --> A
  A --> G[Delivery boundary:<br/>write to disk / execute]

Evidence

Benchmark setup: 173 tasks from CWEval (97) and SecCodeBench (76), covering Python, JavaScript, Java, C and C++. A generation counts as "insecure" if it passes the functional tests but fails at least one security test.

Empirical study (Table 1): six open-weight models, Qwen3.6-35B-A3B, Gemma-4-26B-A4B, DeepSeek-V4-Flash, MiniMax-M3, GLM-5.3-Flash and Kimi-K3, run without assistance.

• Insecure delivery: 13.2–35.6% of functionally valid outputs on CWEval and 42.6–61.3% on SecCodeBench. Qwen is the worst model on CWEval.
• Security mentions don't predict security: in 11 of 12 model–benchmark cells, secure outputs mention "secur-"/"vulnerab-" as often as insecure ones or more. Even among traces that do mention security, 5.4–61.1% still fail security tests.
• The final code exists early: 52.4–99.8% of delivered lines appear verbatim in a fenced block in the reasoning. The draft completes at 45.2–89.1% of the trace, leaving a mean of 203–6,028 reasoning tokens before delivery. DeepSeek leaves the most room for intervention; GLM and Kimi leave the least.
• Case study: a Qwen trace on a URL-building task listed # and ? as invalid characters, then validated only for / and the empty string. It delivered SSRF-vulnerable code (CWE-918), with evil.com# as the bypass.

Self-repair results: beyond the headline rates, the claims of lower token cost than post-hoc repair and smaller regressions under false warnings come without figures in the fetched text.

How to use it

The fetched text gives no repository, commands or release. Deployment as described requires:

• control of the serving stack, so you can pause decoding, splice tokens into the stream and resume from the prefix;
• access to the full raw reasoning stream.

Limits

• Closed APIs are out of reach. The authors excluded GPT and Gemini because those APIs return only thought summaries. The approach effectively assumes self-hosted open-weight models.
• Small-margin models gain less. Models that finish their draft late (GLM, Kimi: about 200–250 post-draft tokens) leave little trace in which a warning can take effect.
• Truncated page. The fetched text stops at the start of the method section. It is missing:
• the ruleset and analyzer used (Semgrep and Bandit are named only as examples of standard analyzers);
• the names of the two self-repair models;
• latency overhead;
• false-positive rates;
• token-cost numbers.

Read the full paper for these before adopting.

• Coverage depends on the rules. Detection is rule-based, so vulnerabilities outside the ruleset are not covered.