SecReason: Vulnerability Detection and In-Stream Self-Repair for LLM Code Generation

What it claims

Reasoning LLMs write most of their final code inside the chain of thought, well before they deliver it. SecReason uses that gap. It runs a static-analysis ruleset over the streamed draft and inserts warnings so the model repairs the code in the same trace. Two parts of the paper go beyond the catalog summary:

• An empirical study of six open-weight reasoning models on 173 tasks. It shows that the model mentioning security in its reasoning does not predict secure output.
• The eval setting itself. The headline gain (from below 15% to 72.8%–98.1% secure and functionally correct) is measured only on tasks where unassisted generation was already vulnerable, uses a "corpus-calibrated" ruleset, and covers only two locally deployed models.

How it works

SecReason sits at the inference serving layer and does not change model weights. The paper names three design problems:

1. When to scan. Static analyzers such as Semgrep and Bandit reject partial code. Waiting until the code fence closes is too late, because later statements may already depend on the flaw. SecReason parses the stream incrementally and triggers a scan on each completed statement inside an open fence.
2. What to rescan. Rescanning everything per statement stalls the stream, while checking one statement alone misses taint flows such as SQL or command injection. The answer is two tracks: - a statement-local track that runs on each statement as it is emitted; - a context-dependent track that traces dataflow across the code accumulated so far.
3. How to feed back. Structured findings (rule ID, line) are turned into a short natural-language warning, for example "[Security] CWE-918 SSRF: subdomain_prefix flows into the URL unchecked." The warning is inserted mid-stream and the task prompt is not changed. Generation then resumes. Aborting and re-prompting in a new turn would throw away the trace and force a full regeneration.

The model decides how to respond. It fixes the draft if the alarm is real or checks and keeps the code if it is false. A false alarm costs a few reasoning tokens instead of a full regeneration.

flowchart TD
  A[Reasoning stream] --> B{Statement completed in fence?}
  B -- no --> A
  B -- yes --> C[Local rule check]
  B -- yes --> D[Cross-statement dataflow]
  C --> E{Rule fires?}
  D --> E
  E -- no --> A
  E -- yes --> F[Pause, insert warning]
  F --> G[Resume decoding: repair or verify]
  G --> A
  A --> H[Delivered code]

Evidence

Setup. The models are Qwen3.6-35B-A3B, Gemma-4-26B-A4B, DeepSeek-V4-Flash, MiniMax-M3, GLM-5.3-Flash and Kimi-K3. Tasks are 97 from CWEval and 76 from SecCodeBench, in Python, JavaScript, Java, C and C++. A generation counts as insecure if it passes the functional tests but fails at least one security test.

Baseline insecurity. Among functionally valid outputs, 13.2%–35.6% are insecure on CWEval and 42.6%–61.3% on SecCodeBench. GLM is lowest on both benchmarks and Qwen or Gemma is highest.

Security mentions don't help.

• The share of secure outputs whose trace mentions "secur-" or "vulnerab-" is equal to or higher than the share for insecure outputs in 11 of 12 model-benchmark cells, by up to 29 points.
• Among traces that do mention these terms, 5.4%–61.1% still deliver insecure code.
• The authors call this lexicon-based and correlational only.

Drafts form early.

• 52.4%–99.8% of delivered lines appear verbatim in a fenced block in the reasoning. Five of the six models exceed 73% on both benchmarks; Kimi-K3 is lowest at 52.4% and 68.8%.
• The best-matching draft closes at 45.2%–89.1% of the trace.
• Mean post-draft tokens range from 203 to 6,028, with DeepSeek highest at 6,028 and 5,147.

Repair. Streaming detection was evaluated by replaying the stored generations of all six models. Live in-stream repair ran on two models. Against post-hoc repair, it:

• matched or beat it in every benchmark/model combination;
• used fewer tokens;
• caused smaller functional regressions when rules misfired on secure code.

The excerpt gives no per-model repair numbers or token counts.

Limits

• Needs the full reasoning stream. Closed APIs that return only thought summaries (GPT, Gemini) were excluded, so the approach needs an open-weight model you serve yourself.
• Ruleset dependence. Headline numbers use a corpus-calibrated ruleset. How rules perform out of the box on other codebases is not shown in the excerpt.
• Narrow evaluation subset. Repair was tested on two models and only on tasks that were vulnerable at baseline.
• Truncated page. The fetched text stops at the start of the methodology section. Rule details, latency overhead, ablations and any code or release information were not available, so check the full paper before adopting.