Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

What it claims

On shared LLM serving endpoints, a model name is not a stable measurement instrument, and the paper's added value over the bare finding is the set of preregistered follow-ups that rule out the obvious fixes. Beyond the headline gate failures (Spearman 0.400 against a required 0.90; byte-identical next-day replay 0.78 against 0.99), the study establishes that waiting, switching providers, scaling sample size, and reading the platform's own metadata all fail to recover stability, and that self-hosting only helps under no load.

Evidence

The four attribution experiments are the reason to read this over the summary:

• Waiting doesn't help. Same-day replay agreed at 0.805 versus 0.800 cross-day, replicated over five further days (one day's own floor lower still). The instability is a within-snapshot, within-afternoon floor, not slow drift.
• Providers don't help. Four providers in three jurisdictions share the floor, medians 0.74 to 0.88, and none of the exposed metadata fields predicted which.
• Self-hosting helps only when idle. Batch-invariant kernels restored stability on a quiet server, but concurrent load raised disagreement 8.4-fold, back to shared-endpoint magnitude.
• Sampling doesn't help. A simulated 748,000-call design passed 0 of 500 times; metric substitution also failed on the tested grid.
• system_fingerprint is not diagnostic. It did not predict stability, in three distinct failure modes.
• The readout is not a magnitude meter. On constructed errors with known gaps, separation tracked error type, not error size (3,060 constructed-error judgments).

The 52,988 attempts are audit volume; the actual statistical base is 31 valid task groups, 100 replay pairs, and ten windows per supplementary arm. The measured object is a two-model arithmetic pipeline: a generator writes four step-by-step candidate solutions, and the observer reads a prefix truncated at 25/50/75/100% of the visible steps. Correctness is decided by exact rational comparison, never by a model. The task family was constructed to be a worst case (tiny candidate gaps).

How to use it

The single actionable rule: run a pilot before freezing any eval gate on an LLM judge. The authors state a pilot at ~2% of the study's call volume (about a thousand calls) would have exposed both unreachable gates in advance. Adopt the pilot–simulate–freeze pipeline: measure the noise floor and gap distribution, run the criterion's operating characteristic, then set the threshold. Treat snapshot identity as mandatory metadata alongside temperature and seed, and report engineering-layer success (delivery, schema, hashes all sat at ceiling here) separately from measurement-layer reliability.

Limits

Claims are scoped to the tested configurations; the paper explicitly does not assert a universal law and says nothing about model internals or any provider's service quality. The task family manufactured its own worst case (candidate gaps seven orders of magnitude below the noise floor), so real evaluations with larger candidate gaps may fare better — the transferable product is the discipline, not the specific floor numbers. Only the abstract, introduction, related work, and the opening of the setup section are in the fetched text; the eight design rules, the L0–L2 ladder details, and the checklist itself (§9, Appendix D) are referenced but not shown.