LLM-based Code Auditing by Inferring and Evaluating Developer Assumptions

What it claims

Creed audits one function at a time by asking an LLM to make the developer's implicit security assumptions explicit, and then to find where those assumptions are wrong or break. It was evaluated on 50 recent CVEs across 10 programming languages, 40 of which are semantic flaws. Creed correctly explained the root cause of 39 of them, a recall of 0.78. Against the strongest explanation-based baseline, recall rose by 10 percentage points and overhead (reports per expected vulnerability) fell from 11.3 to 7.9, about 30%.

How it works

The whole pipeline runs as a single prompt that returns a structured result. The condensed prompt is in Fig. 2; the verbatim prompt and output schema are said to be in the artifact. The prompt walks the model through five steps:

1. Action identification at two levels. Low-level actions, such as a pointer dereference, catch flaws like null-pointer bugs. The function's overall purpose, such as "cancel a payment", catches domain flaws like missing authorization. The LLM picks the decomposition; no granularity is fixed in advance.
2. Belief inference. Each action can imply several beliefs. For example, "a matching siteId authorizes cancellation" and "siteId is server-side and trusted".
3. Sanity check. The model labels each belief Reasonable or Unreasonable against standard security practice and names the practice it compared against.
4. Violation check. The model derives the concrete conditions under which each belief fails. It does this for every belief, including those already judged unreasonable.
5. Reporting. A finding is emitted along either of two paths:

flowchart LR
  B[Inferred belief] --> S{Reasonable?}
  B --> V{Violation condition triggerable?}
  S -- Unreasonable --> R[Finding: root cause + exploit context]
  V -- Yes --> R
  S -- Reasonable --> V

The two paths catch different kinds of flaw:

• Unreasonable beliefs are flagged as context-independent: any code relying on them is vulnerable. Example: CVE-2026-2010, where site-level authorization was treated as payment-level authorization.
• Reasonable beliefs that get violated yield findings that depend on context. Example: CVE-2026-25222, where a generic error was returned to prevent user enumeration, but the extra hashPassword call creates a timing difference that undoes it.

Evidence

• Benchmark: 50 CVEs randomly sampled from recent CVEfixes entries, spanning 10 languages, with 40 semantic flaws.
• Recall: 0.78 (39/50), the highest of the tools compared. That is +10 pp over the strongest baseline.
• Overhead: 7.9 reports per expected vulnerability, against 11.3 for the strongest baseline.
• Under a tight inspection budget, Creed's recall exceeds what any baseline reaches with unlimited inspection effort.
• Non-pattern cases: Creed detected 15 of the 21 cases that cannot be expressed as general static-analysis patterns. It also leads on memory-related vulnerabilities.
• How recall is scored: a report counts if its explanation is consistent with the CVE description, the patch commit message and the patch diff.

The fetched text does not name the baselines or the LLM that was used.

Limits

• Precision is not measured automatically. Many reports beyond the expected one "seem reasonable" to the authors. Their only precision check is a manual review of the first report per function. Expect to triage roughly 8 reports per real bug.
• Function-level scope. A violation condition may already be mitigated by upstream code. Creed only surfaces these conditions for a human to verify; it does not resolve them.
• Small evaluation. The benchmark is 50 CVEs, and the authors chose the scoring criteria themselves.
• Truncated source. The fetched text cuts off in §2.4. Results tables, cost figures, the model choice and any limitations section the authors wrote were not available here.