Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

What it is

An eBay Inc. benchmark (arXiv:2608.27424, Aug 2026) of three static ML-artifact scanners at fixed versions — ModelScan 0.8.6, ModelAudit 0.2.52, Fickling 0.1.12 — against a frozen synthetic corpus of 170 Pickle/PyTorch artifacts in 145 families. The summary carries the headline coverage gap; what it omits is the false-positive story and the ensemble result, which point away from "combine everything" toward a specific per-tool tradeoff.

The tradeoff the summary inverts

The summary says tool combination beats any single scanner. The paper's own numbers complicate that:

• ModelAudit alone detects all 70/70 malicious families and recovers all 48/48 malicious families where ModelScan is incomplete — identical malicious coverage to the full three-scanner union (Table 4). Adding scanners to ModelAudit changes neither its TP set nor its FP set.
• Fickling adds zero unique true positives and zero unique false positives beyond the ModelAudit∪ModelScan union. In this corpus it is pure redundancy.

So the value of combination is not incremental detection — it is decision availability under a scanner's failure, a narrower claim than "better detection."

The false-positive cost

The scanners diverge sharply on benign specimens (Table 3):

ModelAudit's 100% malicious coverage comes with flagging 60 of 65 benign families. ModelScan's perfect conditional F1 comes with flagging nothing benign — but only because it leaves 78 families (including 20 benign) incomplete.

The paper is careful about interpreting these FPs: it separates specimen maliciousness (does this artifact implement malicious behavior?) from deserialization capability (does it contain a primitive a conservative policy could deem executable?). The confusion matrices score the former. A scanner that flags a benign custom-serialization construct because it could execute is counted as a false positive here, but the warning may still be operationally defensible. Treat the 92.3% FPR as disagreement-with-ground-truth, not as noise.

Corpus composition

135 labeled families: 70 malicious (60 unsafe-deserialization, 10 polyglot serialization), 65 benign (50 controls, 15 checkpoints). 10 malformed/truncated families carry no ground truth and are excluded from TP/FP/TN/FN. Formats SafeTensors, ONNX, GGUF are explicitly out of scope — findings apply to Pickle/PyTorch only. Scanner timeouts were 60s (ModelScan), 10s (ModelAudit), 5s (Fickling); ModelScan's incompleteness is partly a timeout-under-load behavior.

Caveats

• The 48/48 malicious recovery does not generalize to benign incompletes: for the 20 benign families ModelScan left incomplete, ModelAudit "recovers" them as false positives. Recovery ≠ correctness.
• Synthetic, deterministic corpus, single frozen run (510 executions); no real-world artifact distribution. The rename-robustness results (15 comparisons) are truncated in the fetched text.