Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

What it claims

nDCG on benchmark qrels fails per query in three ways: it saturates (every reranker scores 1), floors (every reranker scores 0), or compresses (half or more of reranker pairs tie). Across 14 rerankers, one of these happens on 45.6% of NanoBEIR queries and 30.7% of BRIGHT queries. RCP replaces the qrels with dense LLM-derived gains. The pool is labeled once, after which any reranker can be scored with no further judge calls. On NanoBEIR, the share of reranker pairs that a paired t-test separates significantly rises from 34.1% (qrel-nDCG) to 63.5%.

How it works

flowchart LR
  P[Top-150 pool per query] --> A[Stage A: listwise BT tournament]
  P --> B[Stage B: 5 yes/no criteria]
  A -->|within-query order| I[2PL IRT fit]
  B -->|shared criterion params| I
  I --> G[Calibrated gain g in 0..1]
  G --> M[RCP-nDCG]

• Stage A. The LLM judge scores windows of 10 documents, giving each a score in [−5, +5] that the prompt frames as logits. Only the pairwise soft preferences σ(sᵢ−sᵢ′) are kept, and Bradley–Terry scores are fit to them by cross-entropy. A three-phase window schedule concentrates comparisons on uncertain ranks.
• Stage B. Each document is rated on its own against five criteria of rising stringency: Topical Relevance, Information Utility, Entity/Detail Match, Direct Answer, and Thorough Treatment.
• Calibration. The 2PL IRT model gives each criterion a discrimination γ and a difficulty β, both shared across queries. Each query gets a scale τⱼ and offset αⱼ, so calibrated ability is θ̃ = τⱼ·θ_BT + αⱼ. The BT scores are held fixed, and the fit runs once per suite and judge. Because the map is affine, it provably preserves each query's Stage A order. The ideal ranking in RCP-nDCG is therefore the judge's own Stage A order.
• Gain. The gain is the γ-weighted mean of the criterion pass probabilities. On NanoBEIR it is 0.22 at C1's difficulty and 0.69 at C4's. 79.3% of pool documents fall below C1.
• Baseline. Count-nDCG uses each document's pass share as its gain, so documents with equal shares tie. RCP reduces to Count-nDCG when all discriminations are equal.

Evidence

• Benchmarks. 13 NanoBEIR datasets (649 queries), 12 BRIGHT tasks (1,366 queries), TREC-DL 2019 and 2020 (43 and 54 queries), and ViDoRe v3 (2,419 questions, read as OCR text).
• Rerankers. 14 rerankers from Cohere, Contextual AI, Qwen3 Embedding, Voyage AI, Jina AI, and ZeroEntropy.
• Human reference. 46 paid contractors, each with at least a BSc in the annotated subject, graded documents blind. This produced 7,080 grades on 2,268 documents across 311 contests, with 3 annotators per contest.
• RCP gains reach AUC 0.910 at ranking useful documents above non-useful ones, against 0.651 for the qrels.
• Correlation between a query's mean score and its mean human grade rises from 0.538 to 0.795 with calibration.
• Reranker comparisons.
• Where exactly one metric agrees with the annotators' preferred reranker, that metric is RCP-nDCG in 72.4% of 185 contests (chance is about 53%).
• Most contests were deliberately chosen as cases where the two metrics disagree.
• 22 contests produced no verdict.
• TREC-DL. With either of two judges, RCP-nDCG agrees with the NIST grades on every reranker pair those grades separate significantly.

How to use it

Code, the calibrated labels, the LLM judgments, and the 7,080 human grades are released at https://github.com/cohere-ai/rcp-ndcg. The fetched text gives no licence or install commands.

• Using the released labels. They score any model that reorders the authors' candidate pools. Those pools are the top 150 per query, built by Reciprocal Rank Fusion of Cohere Embed v4, Octen-Embedding-8B, and BM25. Positive qrels the retrievers missed were injected into the pools.
• Judges. All are open-weight:
• Primary: Qwen3.5-397B-NVFP4 (17B active).
• Second judge: GPT-OSS-120B (5.1B active).
• TREC-DL: Qwen3.6-27B replaces the primary judge.

Limits

• Pool coverage. The labels only cover documents in the released pools. If a retriever surfaces documents outside them, those documents need new judging.
• Rubric scope. The rubric targets general-purpose retrieval, and the authors say other applications may need other criteria. In their TREC-DL ablation of 15 rubric variants, no significantly separated pair reversed, except when a rubric ignored the query or placed all criteria at one end of the scale.
• Language and qrels. NanoBEIR and BRIGHT are English-only with binary qrels.
• Inherited judge order. Within a query, the metric's ideal ranking is the LLM judge's order.
• Possible bias to check. The authors are at Cohere. Cohere models appear among both the pool retrievers and the evaluated rerankers.
• Truncated fetch. The fetched text stops at Section 4.1, so per-reranker results, judge-agreement details, and cost figures were not seen.