Skip to content

Back to the catalog

GitHub’s model for secret detection: Finding more secrets with less noise
github.blogarticle

GitHub’s model for secret detection: Finding more secrets with less noise

GitHub describes replacing an LLM-based generic secret detection pipeline with a two-stage system. High-recall regex generates candidates, then a fine-tuned ModernBERT classifier scores each one using its file path, secret type, variable names and surrounding context. Training data mixes product feedback, weak-labelled regex candidates and synthetic hard negatives, with splits grouped by repository, value or example. The model is exported to ONNX with 8-bit quantization. Reported results against the LLM pipeline: 13% higher precision, 19% higher recall, 11.5% higher owner-weighted acceptance in an A/B test, 86% lower median post-push scan time and over 98% lower serving cost. It is a concrete case for using a small fine-tuned encoder instead of an LLM on a narrow, measurable classification task.

TL;DR GitHub swaps an LLM secret-scanning pipeline for regex plus a fine-tuned, 8-bit-quantized ModernBERT classifier, with better precision and recall and about 98% lower serving cost.
Open original ↗
Also seen from another angle🛡️ Security →
Document

What it claims

GitHub replaced its LLM-based generic secret detector with a regex-plus-ModernBERT pipeline. The page adds three things the summary does not. The speedup is what made generic secret detection practical in push protection, so findings now come back before a push is accepted, not only after commit. The quality numbers come from a specific two-stage evaluation. The page also describes the data and threshold procedure behind them.

How it works

  • Regex as the recall ceiling. The model only filters what the regex stage proposes. It does not find new secrets. The page frames regex as the right tool for candidate generation and the classifier as the tool for telling "looks like a secret" from "is likely a real secret."
  • Bounded context. Each input is the candidate value, file path, secret type and a bounded window of surrounding context. The page does not give the window size.
  • Hard negatives targeted explicitly. The synthetic data covers documentation secrets, test fixtures, public keys, checksums, structured lookalikes and near-miss pairs. Every synthetic row passes deterministic validation plus a model-based quality review before it is included.
  • Threshold procedure. The alert threshold is chosen in offline evaluation, then checked on production-like traffic before broader rollout. The stated objective is to preserve or improve recall while cutting false positives and serving cost.

Evidence

  • Offline: on post-push scanning, evaluated on the same public traffic as the previous LLM pipeline, precision was 13% higher and recall 19% higher.
  • Online: in a customer A/B test against the LLM pipeline, repository owner-weighted acceptance rose 11.5%. Acceptance is weighted per owner so that customers who bulk-resolve large volumes of alerts do not skew a simple average. This is worth copying if you measure alert acceptance yourself.
  • Speed: median post-push scan execution time fell 86%. Serving cost fell by more than 98% compared with remote LLM calls.

Limits

  • The page does not state the model size, the ModernBERT variant, the context length, the dataset sizes, the chosen threshold, or absolute precision and recall. All quality figures are relative to the previous LLM pipeline, not to regex alone or any public benchmark.
  • No weights, code or data are released. This is a design reference, not something you can adopt directly.
  • Recall is bounded by the regex candidate stage.
  • Some training labels are weak labels from regex candidates.
  • Stated future work includes a production feedback loop for new false positives and false negatives, and a single-step detection-plus-extraction architecture. The current two-stage design may change.

Organisations named here

Technologies named here

Techniques named here

  • Two-stage pipeline
  • Hard-negative training
  • Synthetic data generation for sparse cases
  • Grouped data splitting
  • 8-bit quantization with ONNX export
  • Owner-weighted acceptance

Related in training