Where AI Vulnerability Discovery Harnesses Go Quiet: Structural False Negatives and Hidden Vulnerabilities

What it is

Gadi Evron of Knostic (arXiv 2610.29379v1, 21 October 2026) catalogues 13 deterministic pipeline stages where LLM vulnerability scanners lose real, reachable bugs before or after the model runs. The study starts with the two harnesses the authors maintain, Knostic's OpenAnt and raptor, and adds 10 open-source harnesses: Cisco ai-deep-sast, OpenAI Codex Security, Anthropic ccsr (Claude Code Security Review), RepoAudit, iris, Trail of Bits skills, Vercel deepsec, Visa VV AH, Capital One VulnHunter and Rivian ai-sast. Each is examined at a pinned commit. An empty report cannot distinguish clean code, a model miss and a harness drop.

How it works

The paper's central claim is that rules written to describe typical code ("Python files end in .py") end up deciding what gets analyzed. Anything that does not match is treated as absent or "no vulnerability". The fetched text covers the first five of the 13 drop points:

1. Unrecognized file types: files with no extension or an unknown one are never read, in full scans or in PR scans. Shown by OpenAnt, ai-deep-sast, Codex Security, RepoAudit, ToB skills and deepsec.
2. Argument count mismatch: calls are linked only when argument and parameter counts are equal, so optional or variadic calls are not followed.
3. Built-in name conflict: a project function named like a built-in looks uncalled and is dropped.
4. Unrecognized entry points: framework-invoked code such as background tasks, cloud function handlers and event listeners is treated as dead. One cause is a query that shows the model only functions called by the project's tests.
5. Overridden findings: a correct new finding loses to an older stored verdict, or one step writes a field the next step does not read.

OpenAnt's concrete trace: openant parse → scan_repository → parse_repository → detect_languages → RepositoryScanner.scan_directory → if not is_source_file(entry.name): return. A no-extension file exits there. Renaming it .py restores analysis. The authors show one file dropped by six harnesses before the model saw it, five without notice.

Preconditions / Affected

Two settings are covered. With no attacker, ordinary code is dropped and the report looks clean. With a hostile author who controls file names, paths, comments and contents, the drops can hide a bug on purpose. That attacker cannot change the harness, its configuration, cache or prompts. They may place a file outside a PR diff, and so outside a diff-mode scan. The paper makes no claim about how common each drop is.

What to do

The fetched text ends partway through drop 5, before the recommendations in Section 9, so the paper's design principles and tests are not available here. What the text does give is the audit rule for deciding whether a drop is silent:

flowchart TD
  A[Item dropped] --> B{On a deliverable? report, SARIF, PR comment, exit status, documented output file}
  B -- no, stderr or logs only --> S[Silent]
  B -- yes --> C{Names the item, or a skip count moved and ties to it?}
  C -- no, only a smaller total --> S
  C -- yes --> D[Disclosed]

To test your own harness, plant a vulnerable file with no extension and compare it to the same file renamed to a known extension. Then inspect every deliverable for the file name or a moved skip count.

Caveats

The authors maintain two of the 12 harnesses and use them for most examples. Findings rest on source reading at pinned commits, except runtime checks run without the model. Two model-dependent results used gpt-6-astra: raptor's corroboration gate (six runs) and a planted-comment case (three trials at temperature 0.7). Some OpenAnt runs used an earlier commit, c81c0e1, with the study pin at b9d5d2f. The authors do not claim the list of 13 is exhaustive.