HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation

What it is

HardEval is a public benchmark (code, predictions and per-rule results at github.com/manoharvellala/Hardeval) from Old Dominion University that tests whether an LLM can write a Bash script that makes a live RHEL 8 family host pass a specific DISA STIG check. The paper is arXiv 2610.23685, dated 16 October 2026. The best result is Claude Opus 4.7 at 111/137 (81.0%) on the combined server-safe bucket, followed by Claude Opus 4.8 at 106/137 (77.4%).

How it works

• Task source: An OpenSCAP scan of an unhardened RHEL 8 host (STIG profile, SCAP Security Guide 0.1.81) gives 233 failing rules. Of these, 215 have a reference Bash remediation and 200 are automatable. A fixed cross-model manifest of 151 verified-applicable rules (137 excluding SSH) is graded identically for every model.
• Prompts: Claude Opus 4.8 writes each prompt from the rule's title, severity, rationale and description, never the reference script. Target values such as "15 characters" are kept. Paths, directive names and sysctl keys are removed. A lexical-leakage scan flagged 2 of 215 prompts, both judged benign.
• Grading: The harness gives one shot, with temperature 0, no tools and no retries. Each configuration runs on its own AlmaLinux 8 VM (OpenSCAP 1.3.14) reset to the failing baseline. reboot, shutdown, poweroff and systemctl reboot are neutralized through a sanitized PATH. The verdict comes from the parsed rule result, and only a literal pass counts, not notapplicable.
• Models: Fourteen configurations: seven closed (Claude Opus 4.8, 4.7 and 4.5, Sonnet 5, Sonnet 4.5, Haiku 4.5, GPT-4o) and seven open-weight. The open-weight set includes DeepSeek-Coder-33B, CodeLlama 34B and 7B, Qwen2.5-Coder 14B and 7B, and GLM-4-9B at FP16 and 4-bit Q4_K_M.

What to do

For model selection, especially for self-hosting, the authors suggest using it to find a base model that reaches scanner-verified compliance before investing in fine-tuning or retrieval. The grader ingests a predictions_<model>.jsonl file, so other models can be scored the same way. The text is cut off before the exact invocation, so check the repository.

Caveats

• The benchmark measures satisfaction of the STIG compliance oracle, not security in general. A script can harden the system by another route and still fail. A target-rule pass does not rule out collateral effects.
• Scripts accumulate on the VM during a run. The aggregate is exact, but credit for one rule may not come from the script written for that task.
• The manifest leaves out reboot-persistent kernel rules, hazardous access-severing rules and GUI or hardware-dependent controls.
• The harness assumes non-adversarial models. A scan of 6,494 generated scripts found no reference to the scanner or grading artifacts. A run that disabled scoring would be excluded as invalid.
• Multi-run Wilson confidence intervals preserve the ordering for only four open-weight configurations.
• The authors report that model ordering held when GPT-4o re-authored the prompts and in a zero-code port to Ubuntu 24.04. The fetched text does not include those results.
• Per-model scores beyond the top two, and the quantization cost for GLM-4-9B, are not in the fetched text.
• The common failure is a scanner-specific value or mechanism mismatch, not shell syntax errors.