AI code security harnesses compared

What it is

A Semgrep blog post surveying open-source "harnesses" that point an LLM at a codebase to find vulnerabilities, sorted into three categories and mapped to concrete adoption decisions. The value beyond the summary is the routing: which tool fits which situation, plus specific design facts (static vs. dynamic reasoning, same vs. separate model for discovery/validation) that determine whether a tool suits your workflow.

How it works — the three categories

• LLM-led exploitgen: drives toward a crashing end-state as a vulnerability oracle — "a new form of fuzzing." Hardest to actually use because model guardrails block exploit generation (you need trusted/verified access), and each is narrowly scoped to a few vuln types with reliable exploit conditions. Uses the same model to both discover and generate the exploit.
• LLM-skill-boosting: adds skills so the LLM reasons like a human researcher. Includes Cloudflare security-audit-skill (6-phase, adversarial validation), Trail of Bits skills (~40 plugins), CapitalOne vulnhunter (find/fix/verify loop), Google mantis (~15 skills).
• SAST+LLM hybrids: feed deterministic analysis (Semgrep, CodeQL) to the LLM to guide the search. Includes raptor, Vercel deepsec, Cisco ai-deep-sast, Visa VVAH.

What to do — pick by situation

• Vuln hunter, want breadth + operational appetite: raptor (scanning, dataflow, fuzzing, exploit feasibility, SCA; can compile/execute to validate).
• Vuln hunter, willing to invest and build your own: google/mantis as a starting point.
• Appsec program optimizing time-to-reviewed-fix: Visa VVAH — broad-language (42 languages), 11-stage pipeline with threat modeling, adversarial verify, exploit-chain synthesis, propose-and-validate-the-fix. Accepts a frontier-model dependency; no runnable PoC.
• Already in Claude Code / Codex: install skills — Cloudflare's for near-zero-setup adversarial audit methodology; ToB's for specialist reviewers (native code, crypto constant-time, smart contracts); capitalone/vulnhunter if you're specifically on Claude Code.

Cross-cutting facts worth knowing

• Static vs. dynamic: ai-deep-sast, VVAH, deepsec reason purely statically (no code execution). raptor (and defending-code-harness) compile and execute binaries to validate — closer to real human research.
• Adversarial validation (a second independent agent tries to falsify each finding) is widely adopted for accuracy: Cloudflare Phase 3, VVAH S6, ToB fp-check.
• Patch generation remains uncommon among SAST+LLM hybrids; exploitgen tools produce PoCs and patches.
• Language support is no longer a differentiator — LLMs handle most languages out of the box; the exploitgen bottleneck is the build toolchain, not the language.

Caveats

The author expects no single OSS reference harness soon — the field moves too fast, and several projects are already marked "no external contributions accepted" or unmaintained, so factor maintenance risk into any adoption. The post is also self-interested: Semgrep teases its own Black Hat announcement.