Negative Days in LLM Workflows

What it is

A DIY threat-intelligence workflow — a GitHub Action cron job — that detects security patches on open-source repos by feeding commit diffs and pull-request context to the Claude API, then opens a GitHub issue as an early-warning alert. Motivated by the "negative-days" problem: for public projects, the patch commit and PR are visible on GitHub before the CVE feed publishes. A live demo runs at https://vulnerabilityspoileralert.com.

How it works

The workflow uses a state.json to record the last-checked commit hash, checks all commits since the last run, and passes each to Claude to judge whether it patches a vulnerability. The author deliberately chose a workflow, not an agent (citing Anthropic's "when not to use agents" guidance). The initial version fed only git diff; after false positives/negatives, it added PR context via GitHub's listPu[llRequests]-type API — labels and descriptions carry more signal than short commit messages.

The tuned detection prompt instructs Claude to look for:

• Input validation being added; sanitization of user input
• Injection fixes (SQL, command, XSS, etc.)
• Authentication or authorization fixes
• Buffer overflow / memory-safety fixes
• Cryptographic improvements
• Path-traversal fixes
• Rate limiting or DoS protections
• Security keywords in commit message (CVE, security, vulnerability, fix, patch)
• PR labels indicating security ("security", "vulnerability", "CVE")
• PR description mentioning security issues, CVE identifiers, or vulnerability details

Why the window exists

The React2Shell case illustrates the gap concretely:

timeline
    10:00 PM : Patch commit pushed (public forked repo)
    11:38 PM : Pull request opened (PRs on public repos can't be private)
    11:40 PM : CVE published to MITRE
    12:00 AM : MITRE hourly cadence lets most feeds propagate

Monitoring GitHub activity directly yields ~2 hours' warning over CVE intel feeds — and React2Shell was a well-run, Meta-coordinated disclosure. Poorly-managed projects widen the gap further. The author also flags "never-days": vulnerabilities quietly patched but never assigned a CVE, so users on vulnerable versions have no signal to upgrade — especially messy when a large CVE is followed by minor CVEs and silent hardening commits.

Caveats

The page states the diff-only approach produced false positives and negatives, fixed only partially by adding PR context — so alerts still require triage. The React2Shell timeline is a best-case, tightly-coordinated disclosure; the 2-hour figure is not guaranteed for other projects. The fetched text cuts off mid-sentence at the listPu GitHub API detail, so implementation specifics beyond the prompt are incomplete here.