Skip to content

Back to the catalog

HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models
arxiv.orgpaper

HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

HoF-Bench is a benchmark of 95 real, publicly disclosed CVEs originally found by AISLE's LLM-based vulnerability analyzer (280+ CVEs across 78 projects including OpenSSL, curl, and GnuTLS), repackaged into eight repositories pinned at vulnerable commits. Analyzers get source/target-file scope but no CVE ID, description, fix, or expected mechanism, and a detector-blinded frontier-model judge only credits findings that match the same code path, root cause, attack condition, and impact. Testing ten non-frontier backbones (open-weight 21B-284B and proprietary "flash"-tier models) in a fixed multi-pass scaffold, a minimal analyzer rediscovers up to 68% of the CVEs, with misses concentrated in C infrastructure code — showing small/cheap models can approximate frontier-level vulnerability discovery on known bug classes.

TL;DR New benchmark (HoF-Bench) shows non-frontier LLMs can rediscover up to 68% of real AI-found CVEs in OpenSSL/curl/GnuTLS-class C codebases under a strict, blinded grading protocol.
Open original ↗
Document

What it is

HoF-Bench v1, a source-conditioned vulnerability-rediscovery benchmark from AISLE, holding 95 public CVEs across eight repositories pinned at vulnerable commits: OpenSSL (24), OpenEMR (12), curl (10), pearweb (9), GnuTLS (8), Apache httpd (5), FOGProject (5), and WeKan. It spans 39 CWEs — top classes are XSS (CWE-79, 9), SQLi (CWE-89, 8), incorrect authorization (CWE-863, 8), improper access control (CWE-284, 7), and NULL-pointer dereference (CWE-476, 6). 31 of the 95 are multi-file cases. Dataset: https://huggingface.co/datasets/aisleinc/HoF-Bench.

The operational findings the summary omits

The headline "up to 68%" is a union of four passes, not a single run — this is the actionable point. No single scan gets there:

  • Repeated passes are the biggest lever. Averaged over pass subsets, four passes find ~12–20 more CVEs per model than one pass. Six of ten model families cleared 50/95 in one four-pass condition; the best was 65.
  • Model diversity beats repetition of one model. Spending the same pass budget across different models improves rediscovery because their blind spots only partially overlap.
  • The two "obvious" pipeline additions barely help. Generated context (a security briefing plus up to three repo searches) cuts universal misses but its pooled recall gain is small and uncertain while candidate volume rises. Going from one triage round to three-plus-arbiter shifts strict outcomes by only a few percent.
  • Difficulty tracks language, not bug class. With context, 21 CVEs are found by every model and 11 by none; 18 of the 21 universal finds are JavaScript or PHP, and 9 of the 11 universal misses are C. DoS/memory-safety cases concentrate in C, SQLi in PHP, authz failures in the web apps.

Detector panel and cost

Ten non-frontier detectors, priced \$0.03–\$1.50 per million input tokens at July 2026: five open-weight MoE models (GPT-OSS 20B/120B, DeepSeek V4 Flash 284B, Qwen 3.6 35B A3B, Mistral Small 4; 3–13B active params, MIT/Apache) and five proprietary flash-tier (GPT-5.4 nano/mini, GPT-5.6 luna, Gemini 2.5/3.5 Flash). A frontier model (GPT-5.6 terra, \$2.50/\$15.00) appears only as the blinded judge, never as a detector.

Caveats the paper states

  • Task supplies 1–7 target files including the primary vulnerable file — a "substantial localization oracle," so this measures analysis/rediscovery, not repo-wide localization.
  • Recall-only metric: every task is positive, unmatched findings are not labeled, so the authors decline to call them false positives.
  • No cross-task semantic assignment yet (listed as a limitation).

Technologies named here

Techniques named here

  • LLM-based vulnerability detection/rediscovery

Related in AI Security