Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions

What it is

UNDERSPECBENCH's design and methodology, beneath the headline numbers the summary already gives. The benchmark turns instruction underspecification into a controlled variable: 69 incident-grounded DevOps task families, each instantiated across a 4×4×2 = 32-variant matrix (32 × 69 = 2,208 prompts), holding environment, tools, and oracle fixed so any behavior change is attributable to the instruction alone. The fully-specified S0B0R0 baseline is hand-built; the other 31 variants are derived mechanically by slot substitution.

The three axes (mapped to NIST ABAC)

The perturbation axes are not arbitrary — they map to three of the four request attributes NIST's Attribute-Based Access Control model requires for an authorization decision (the fourth, subject, is held fixed as the user-authorized agent):

• (S) Intent Clarity, S0–S3 — how explicitly the action and operating environment are stated; higher levels strip context and inject production-like cues (ABAC action).
• (B) Target Certainty, B0–B3 — how uniquely the object is identified; higher levels use under-determined references ("the old one," "the stale config") matching several candidates (ABAC object).
• (R) Blast Radius, R0/R1 — contained low-impact surface vs. shared/production surface where a mistake propagates and cannot be undone (ABAC environment).

The oracle is deterministic and per-task: it diffs before/after environment state (file diffs, command logs, structured service-side action traces) against an intended set (objects that should change) and an over-scope set (everything that must stay untouched).

Details the summary omits

• Exact configurations tested (five): OpenCode+Haiku-4.5, OpenCode+Codex-5.1-mini, OpenCode+DeepSeek-v4, Claude Code+Haiku-4.5, Codex+Codex-5.1-mini.
• The motivating pressure stat: Anthropic reports Claude Code users approve 93% of permission prompts, motivating auto-mode as a response to approval fatigue — the paper's case for why underspecified-but-benign instructions matter now.
• Grounding anchors: destructive-data tasks tie to the 2017 GitLab production-database deletion; container-hardening tasks to CVE-2019-5736; the framing incident is the 2026 PocketOS case where an agent, given a staging-scoped task whose scope wasn't enforced in the instruction, acted on production and deleted the production database plus volume-level backups.
• Taxonomy: four capability domains (change intake/source config; build & verification; release & artifact supply chain; runtime & reliability ops) over nine control surfaces, derived from SLSA supply-chain stages and DORA capabilities.

Caveats

Each task family is a simplified abstraction of a real operation, explicitly not an adversarial trap — the benchmark measures behavior under benign ambiguity, not malicious or overtly dangerous requests, which is a deliberate scope limit distinguishing it from prompt-injection and harmful-action studies.