Skip to content

Back to the catalog

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
arxiv.orgpaper

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

Speculative Uncertainty (SU) repurposes speculative decoding in reverse: a small open-weight draft model scores an agent's already-generated trajectory in a single forward pass, producing a failure-likelihood signal from output tokens alone without needing logits, weights, or repeated sampling. Phase-aware features separate reasoning and action spans and are calibrated against a verifiable objective to drive a pre-execution veto gate. On software engineering agents built on Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, the gate cuts execution error rate by 6-8 percentage points and token cost by 14-19%, transferring to out-of-distribution benchmarks without retraining.

TL;DR A small draft model scores a coding agent's trajectory to predict failure before execution, cutting error rate 6-8pp and cost 14-19% on Qwen3-Coder-480B and Claude 3.5 Sonnet.
Open original ↗
Document

What it claims

Speculative Uncertainty (SU) recovers a failure-prediction signal for a black-box coding agent from output tokens alone, reaching .77 AUROC on SWE-Bench Verified versus .57 for the agent's own verbalized confidence and .68–.70 for white-box mean-log-prob baselines, approaching the white-box HTC ceiling of .82 without any access to the agent's logits or weights. The draft is a Qwen3-4B model scoring agents built on Qwen3-Coder-480B and Claude 3.5 Sonnet.

How it works

For each token the draft q runs in teacher-forcing mode over the agent's committed tokens, yielding three per-token signals from a single forward pass: speculative surprisal sₜ = −log q(yₜ|·), speculative gap gₜ (the draft's argmax log-prob minus the observed token's), and speculative entropy Hₜ. The justification is the speculative-decoding acceptance rate α = min(1, p/q): if q tracks p, high surprisal implies the hidden p would also reject there, so sₜ proxies the agent's own uncertainty — which is why aligning the draft to the agent matters.

Features are computed separately over reasoning and action spans (the harness message structure labels each). Eight statistics (mean, variance, max, min, skewness, trend, first-10%, last-10%) × 3 signals × 2 phases + 2 span lengths = 50 features, fed to an ℓ1-regularized logistic regression over a lookback window of k=3 recent steps. Trained on SWE-rebench OpenHands trajectories from Qwen3-Coder-480B.

Evidence

  • Phase separation is worth +5–6 AUROC: pooling phases (SU-Uniform) drops .77 → .71, confirming the action-span signal is masked once reasoning tokens are averaged in.
  • Alignment matters, objective barely does: untrained draft only .61 AUROC (near verbalized confidence); SFT and teacher-forced distillation both reach ~.77–.80 and land within 1 point of each other.
  • Veto gate deployment (Table 2): error rate 21%→15% on SWE-Bench Verified (−6pp) and 14%→6% on DA-Code (−8pp); token savings 14–19%; true-veto recall 0.72/0.69 at false-veto rate 0.14/0.16.

Limits

  • SU is a ranker, not a calibrated probability. The authors explicitly relabel the output a "failure-likelihood score." Brier skill score is only .215 even after distillation; they flag closing the reliability gap as required work before any policy that reads the probability value (rather than a threshold) can consume it. Temperature/isotonic recalibration was not applied.
  • Task success drops: the gate costs −5% task success on SWE-Bench Verified (49%→44%) and −1% on DA-Code — the error/cost gains come with vetoing some actions that would have succeeded.
  • Does not beat white-box HTC (.77 vs .82) and is not claimed to; the value is matching the white-box ceiling with black-box access.
  • Verifiable objective here is narrowly "code runs without exceptions," not correctness.

Technologies named here

Techniques named here

Related in coding-agents