GitHub’s model for secret detection: Finding more secrets with less noise

What it is

GitHub's account of replacing an LLM-based generic-secret pipeline in secret scanning with a regex candidate stage plus a fine-tuned ModernBERT classifier. The useful parts for someone building something similar are the design rules and how the results were measured, more than the headline numbers.

How it works

• Regex stays as the deterministic, high-recall first stage. The classifier only decides between "looks like a secret" and "is likely a real secret". It never finds new candidates.
• Each model input is the candidate value, file path, secret type and a bounded window of surrounding context. The page does not give the window size, model size or latency figures.
• The output is a score. The threshold is chosen offline, then validated on production-like traffic before rollout.
• Synthetic hard negatives (docs secrets, fixtures, public keys, checksums, near-miss pairs) go through deterministic validation and model-based quality review before they enter training.
• Rows sharing a repository, candidate value or example are grouped before the train/eval split, so near-duplicates can't leak across it.
• The model is exported to ONNX with 8-bit quantization.

What to do

If you are building or evaluating something similar, the page implies these practices:

• Keep the rules engine for recall and use the model only to suppress false positives.
• Split data by group (repo, value, example), not by row.
• Mine hard negatives deliberately.
• Evaluate in two stages: offline precision and recall against the incumbent on the same traffic, then an A/B test on user behavior.
• Weight acceptance by repository owner. GitHub does this so customers who bulk-resolve many alerts don't distort the average.

Caveats

• All results are relative to GitHub's previous LLM pipeline. No absolute precision, recall, latency or cost figures are given.
• The offline numbers (+13% precision, +19% recall) are for post-push scanning on public traffic. The page says nothing about how recall ground truth was established.
• The 86% scan-time reduction is also post-push. Push protection gets no measured numbers, only the claim that the speedup made it practical.
• Because regex generates the candidates, the classifier cannot recover secrets the regex stage misses. The page says GitHub is investigating single-step detection and extraction models.
• The page presents acceptance as a proxy for precision. It does not say how many accepted alerts were true secrets.