Benchmarking Endor Labs AI SAST: 2.6x more real vulnerabilities found than frontier models | Blog | Endor Labs

What it is

Endor Labs' vendor-run benchmark of its AI SAST product against four pattern-based scanners (Semgrep OSS, OpenGrep, Bearer, CodeQL) and two frontier models prompted directly (Claude Opus 4.7, Codex GPT-5.5). The corpus is 8 open-source projects across Java, Python, JS/TS, C# and Go. The product is generally available.

What the numbers add

Beyond the headline of 192 true positives, the page gives figures that matter more when you are choosing a tool:

Absolute recall for the best tool is still under half of the verified pool. The page's own tradeoff: frontier models give near-zero false positives but miss most issues, and pattern scanners give dependable coverage of well-understood classes like injection but need triage resources.

How it works

Claude and Codex explore a narrow, shifting slice of the repo on each run. On a large enterprise Java codebase, the security-relevant code they examined fell below 10%. Pattern scanners cover every supported file but only find what their rules describe.

Endor's pipeline reuses its SCA call-graph and reachability engine on first-party code:

1. Index the code and build a code graph across services and repos.
2. Add framework and infrastructure context, such as existing protections and internet exposure.
3. Trace dataflow across files and services.
4. Triage: every function is screened, only the interesting ones get deep AI analysis, and findings are re-checked against the code (for example, spotting an existing sanitizer).
5. Classify with CWE, severity scored against application context, and evidence (data flow and call paths).

Findings also come with exploit validation and suggested fixes. They surface via baseline scan, PR check, IDE, CLI, MCP server, and hooks and skills for AI coding agents.

Example finds

The unique finds clustered in access control/IDOR, authentication failures and business logic, which pattern matching handles poorly.

• buffa (Anthropic's Rust protobuf library), CVE-2026-55407: untrusted wire data flows into an unbounded heap allocation reachable from the default decode APIs. A small input causes roughly 22x memory amplification and an OOM kill.
• Shopizer: the product-image delete endpoint deletes by client-supplied ID with no check that the image belongs to the caller's store, so one merchant can delete another's images. The missing check spans controller and service.
• WebGoat: the account-verification check validates the count of submitted answers and that recognized keys are correct, but never requires those keys to exist. Two arbitrary keys pass and both real checks are skipped.

Caveats

• All results are vendor-reported, and the ground truth is built from pooled tool findings plus the vendor's manual review. Vulnerabilities nobody found are not counted, so recall is relative to that pool.
• Each tool is scored only on the languages and projects it supports, so denominators differ by tool.
• AI tools vary between runs. The page reports an aggregate over multiple runs but gives no variance.
• The Claude and Codex runs were plain prompted scans. The page notes Mythos-class models were not tested.
• The page names the corpus only as 8 projects (it does name Shopizer, WebGoat and buffa). It does not describe the prompts given to the frontier models.
• Pricing, scan time and cost per scan are not given.

If you evaluate it, the page's invitation is to run it on your own codebase via a demo. Use your own repos and compare against your current scanner's findings, since the benchmark corpus is small.