HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation

What it claims

HardEval tests whether a language model can write one Bash script that makes a live RHEL 8 system pass a specific DISA STIG OpenSCAP/OVAL check. On a fixed manifest of 137 non-SSH rules, Claude Opus 4.7 solved 111 (81.0%) and Claude Opus 4.8 solved 106 (77.4%). The authors report that failures are rarely shell-syntax errors. Most are scanner-specific mismatches, where the script uses the wrong value or a different mechanism from the one the check expects.

How it works

The tasks come from one OpenSCAP scan of an unhardened RHEL 8 host. The scan used the STIG profile from SCAP Security Guide 0.1.81 (ComplianceAsCode). Of 233 failing rules, 215 have a reference Bash remediation and 200 are automatable; these 200 are the graded tasks. The cross-model manifest is smaller: 151 rules, or 137 without SSH. It leaves out reboot-persistent kernel rules, rules that could cut off access to the host, and GUI- or hardware-dependent controls.

Claude Opus 4.8 wrote the prompts. It saw each rule's title, severity, rationale and description, but never the reference script. Prompts state the objective and any required values, such as a 15-character minimum password length. They withhold file paths, directive names, sysctl keys and commands. A whole-word leakage check against tokens from the reference scripts flagged 2 of 215 prompts, and both were judged benign.

Grading is single-shot, with no retries, no tool access and no inspection of the live system. Each model configuration gets its own AlmaLinux 8 VM, reset to the failing baseline. The grader uses OpenSCAP 1.3.14. Reboot and shutdown commands are neutralised through a sanitised PATH. A task counts as solved only when OpenSCAP reports the literal result pass; notapplicable does not count. A full rescan at the end of each run agreed with the per-rule results in every retained RHEL run.

Evidence

The study covers fourteen configurations, all decoded greedily at temperature 0:

• Closed models: Claude Opus 4.8, 4.7 and 4.5, Sonnet 5, Sonnet 4.5, Haiku 4.5, and GPT-4o.
• Open-weight models: DeepSeek-Coder-33B, CodeLlama-34B and 7B, Qwen2.5-Coder-14B and 7B, and GLM-4-9B at FP16 and 4-bit Q4_K_M.

The Qwen models ran on vLLM on one A100, and the others ran on Ollama. For four open-weight configurations, multi-run Wilson intervals preserve the observed ordering. The authors also re-authored the prompts with GPT-4o and ported the pipeline to Ubuntu 24.04, and they report the model ordering stays stable. The fetched text does not include the per-model scores for the open-weight models or the numbers from these robustness studies.

The closest prior system, SHIELDS, resolves up to 73% of STIG findings. It is not directly comparable: SHIELDS is a multi-agent loop that uses scanner feedback, while HardEval is single-shot.

How to use it

The benchmark, source code, model predictions and per-rule results are at https://github.com/manoharvellala/Hardeval. The prompt-generator system prompt and exemplars are in the same repository. The grader runs separately from inference and takes a predictions_<model>.jsonl file. The fetched text gives no install commands and no licence.

Limits

• Narrow target: HardEval measures whether a script satisfies the compliance check, not whether the system is more secure overall. A script can harden the system another way and still fail.
• Collateral effects unchecked: a passing rule does not show that the script left the rest of the host unharmed.
• Credit attribution: scripts accumulate on one VM during a run, so the aggregate score is exact but credit for a single rule may come from another task's script.
• Non-adversarial assumption: the harness assumes models do not game the grader. A scan of all 6,494 generated scripts found no references to the scanner or grading files. A run in which a script disabled a whole scoring subsystem was excluded as invalid.
• Coverage gaps: the 137-rule manifest excludes reboot-persistent, access-severing and GUI rules, so those controls go untested across models.
• Truncated source: the fetched page stops partway through the setup section, so the full results tables are not covered here.