Skip to content

Back to the catalog

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
arxiv.orgpaper

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

A preregistered audit of whether identical requests to the same-named LLM API return stable outputs over time, finding they largely do not: across 52,988 audited requests, same-window repeat rankings hit Spearman 0.400 against a required 0.90, and next-day byte-identical replays hit 0.78 against a required 0.99. The authors trace the failure to label-mapping bias, candidate gaps far below the instrument's noise floor, and non-deterministic outputs for identical inputs, tested across four providers and self-hosted batch-invariant kernels, concluding a model name on shared infrastructure is not a frozen measurement instrument and that evaluations must characterize instrument noise before freezing any gate on it.

TL;DR Preregistered study shows LLM-as-judge rankings on shared API endpoints are far less stable/reproducible than commonly assumed, undermining eval gates built on them.
Open original ↗
Document

What it claims

LLM-as-judge pipelines assume a hidden invariant: the same request to the same model name reads the same tomorrow. Two preregistered audits, thresholds fixed in advance, both failed at the instrument-validation stage before any actual comparison. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90; byte-identical next-day replays agreed at 0.78 against a required 0.99 — while the execution record (that calls actually completed as specified) sat at ceiling. So the pipeline ran clean; the measurement was unstable.

Evidence

Three mechanisms are named: a label-to-meaning mapping that biased readouts as strongly as the signal itself; candidate quality gaps seven orders of magnitude below the instrument's own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts amplify. Metric substitution and sampling did not repair it on the tested grid.

Preregistered follow-ups bound the cause:

  • Waiting: no help — 0.805 vs 0.800, replicated over five further days.
  • Switching providers: no help — four providers share the floor, medians 0.74 to 0.88, and none of the metadata fields they expose predicted the outcome.
  • Self-hosting on batch-invariant kernels: helped only while the server was quiet, i.e. concurrency reintroduces the noise.
  • Constructed errors with known gaps: the readout's separation tracks error type, not error size — so it does not measure magnitude.

How to use it

The paper's practical output for evaluation builders: a three-level snapshot-identity ladder, eight design rules, and a reporting checklist. Its cheapest takeaway: a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance — so run an instrument-noise pilot before freezing any threshold that a judge will gate on.

Limits

This is an abstract-only arXiv page (v1, submitted 3 Sep 2026); the ladder, rules, and checklist are named but not reproduced here, so the actionable content requires the PDF. All results concern externally measured behaviour on shared serving infrastructure, on the specific providers and grid tested; the self-hosting result suggests determinism is recoverable under controlled, low-concurrency conditions the black-box case cannot guarantee.

Techniques named here

Related in evals