TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge

What it claims

TRIAGE is a pre-registered study of two-tier LLM serving on one NVIDIA A40, with Qwen2.5-7B-Instruct-AWQ as the small tier and Qwen2.5-32B-AWQ as the large tier under vLLM 0.6.3.post1. Its central result is a null. On a 626-item held-out split, the 32B model beat the 7B model by only +2.88 pp (95% CI [−0.80, +6.55]), so escalation could not help. A 66-item pre-launch gate had measured +13.6 pp and passed its 8 pp bar, which is how the null went unnoticed. The paper also proves that any prompt-only router scores AUROC exactly 0.5 on paired evidence-corruption benchmarks. It shows that a hosted semantic judge (Jev) predicts wrong small-model answers well. Those Jev results are post hoc and were added after the pre-registered detection claim failed.

How it works

The small tier answers first. A reliability state is then computed from five fixed-weight components: token uncertainty, resample instability, contradiction, retrieval disagreement, and a lexical cosine evidence-sufficiency score. An allocator weighs eight priced interventions against their GPU-second price. The price is multiplied by a factor that rises with measured device concurrency. The two tiers run as separate vllm serve processes with disjoint KV pools on one GPU.

flowchart LR
  R[Request] --> S[7B answers]
  S --> RS[Reliability state]
  L[Measured concurrency] --> P[Load-dependent price]
  RS --> A{Allocator: value vs price}
  P --> A
  A --> stop
  A --> probe/verify
  A --> E[Escalate to 32B]
  A --> abstain

Thresholds inherited from an earlier API-based system could never fire. The 7B model's uncertainty spans [0.0033, 0.1483], but the escalation threshold was 0.28. They were recalibrated to measured percentiles. Jev (pinned jev-1.13.0, from TypeSafe) is used in four roles: evidence sufficiency, answer support, routing, and image support. Its latency is p50 356–418 ms and p95 447–806 ms, at about $0.00002 per call.

Evidence

• Tier gap by dataset (32B − 7B): MMLU-Pro +11.32 pp is the only gap whose CI excludes zero. GPQA-diamond is −3.76 pp, HotpotQA +2.53, and MATH-500 +0.57. In the 2,504-row headline stage, TRIAGE matched always-small at 0.3578 accuracy, using 4.18 against 3.77 GPU-s per request. Against a 14B mid-tier baseline, the result is inconclusive (+1.13 pp, CI [−3.20, +5.27]). Resolving it would need about 19,600 items.
• Corruption detection on RGB (300 items): Every generation-derived signal sits at chance, with uncertainty at 0.504 and instability at 0.500. RGB negatives are fluent and on-topic. Putting Jev into the existing weighted risk reaches only 0.695, because the evidence term carries 0.15 of the weight.
• RouterBench (test n = 1,708 per pair, API dollars): Triage AUROC is 0.955, 0.945 and 0.898, against 0.599–0.681 for prompt-only.
• For GPT-3.5→GPT-4 (about a 17× price ratio), triage beats random by +8.31 pp at a 50% budget.
• For Mistral-7B→Mixtral-8x7B it is −2.47 pp against random, and for WizardLM-13B→Yi-34B it is −2.09 pp. The extra small call is not worth paying at 2–3× ratios.
• Non-inferiority to always-big holds at about 3 pp, not the 1 pp point estimate.
• Off-policy evaluation: Checked against RouterBench ground truth, the doubly robust estimator has RMSE 0.86–1.26 pp, about half that of IPS. Its CIs are conservative, with 0.98–1.00 coverage.
• HallusionBench (356 pairs): Jev image support reaches AUROC 0.834. It is strongest on tables (0.944) and weakest on math (0.699).
• Escalation price: 0.01415, 0.237 and 0.309 GPU-s at 1, 4 and 10 rps. Small-tier P95 rises by +23.4, +83.5 and +109.3 ms. The 1-rps CI is as wide as its estimate, so the authors give about 11× as a conservative ratio.

Limits

The Jev wrong-answer AUROC of 0.922 on RGB comes from closed-book answers. Passages reached the 7B model only if a verification step ran, so its error rate was 72.8%, including 71% on clean items. The authors state this measures detection, not whether routing on Jev repairs answers, and that repair is unmeasured. Price CIs come from two arrival seeds and understate between-run variation by about 1.8×. The KV-headroom effect is a replicated null: a fresh-host re-run reversed the first step's sign. GSM8K was excluded from RouterBench because its partial scoring inverted the signal. Jev's image input is labelled a prototype by its vendor. The fetched text ends mid-section, and no code or release is mentioned.