BenchMIRT: What are LLM benchmarks actually measuring?

What it claims

BenchMIRT, from Ai2, applies multidimensional Item Response Theory (MIRT) to LLM benchmark results at the level of individual prompts. It estimates which latent capabilities each question actually depends on. The model was fit on results from 100 LLMs across 16 benchmarks (more than 34K questions). Without being told which benchmark measures what, it recovered two stable dimensions: safety and general reasoning. Several safety benchmarks turn out to load mainly on reasoning:

• BBQ aligns more strongly with general reasoning than with safety.
• WMDP aligns more strongly with reasoning than with safety, and stronger reasoning goes with lower WMDP scores.
• HarmBench splits: its copyright items track reasoning, while its standard and contextual items track safety.

How it works

• Starting point. Single-dimensional IRT has already been applied to individual benchmarks, including in Ai2's Fluid Benchmarking work. BenchMIRT extends this to multiple dimensions so that several capabilities can explain performance on the same question.
• Per model. It estimates the model's ability on each latent dimension.
• Per question. It estimates the question's difficulty and its discrimination, meaning how well it separates stronger from weaker models, on each dimension.
• Training data. 6 general-reasoning benchmarks (including MMLU-Pro, GPQA, MATH, BBH) and 10 benchmarks from the Olmo 3 safety suite (including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, XSTest).
• Stability. Repeated from-scratch fits recovered the same two dimensions each time.

Evidence

• Benchmark alignment. Measured as Pearson correlation, across 100 open-weight LLMs, between BenchMIRT ability scores and each benchmark's score. Most reasoning and jailbreak/harmful-content benchmarks aligned with their intended dimension.
• WildJailbreak. Its harmful prompts track safety, while its benign over-refusal prompts track general reasoning. Averaging the two into one score hides that split.
• WMDP sign. The negative relationship with reasoning arises because the benchmark scores refusing, or failing to supply, the dangerous knowledge as the correct response.
• Item subsetting. Questions were ranked by discrimination while keeping a spread of difficulties. Keeping 10% of questions "generally preserved nearly the same" model ranking on the underlying capability as the full set, and keeping 50% "often" matched it more closely. The page gives no exact correlation figures for these.
• Held-out prediction. BenchMIRT predicted whether a model answers an unseen question correctly 79% of the time. A baseline that assumes per-question accuracy equals the model's overall benchmark accuracy got 70%.

How to use it

The post gives no code, package, install command or licence. What it does establish is how to read existing scores:

• BBQ. A low score may reflect reasoning difficulty, not bias behaviour alone.
• WMDP. A low score may reflect a stronger reasoner rather than a less safe model.
• HarmBench and WildJailbreak. Composite scores mix signals. Consider reporting their sub-groups separately (copyright vs. standard/contextual for HarmBench; harmful vs. benign for WildJailbreak).

Limits

• Model recency. All 100 models were released by March 2025, so behaviour on newer model generations is untested.
• Benchmark mix. The dimensions found depend on the benchmarks supplied. Safety and reasoning came out of this particular set of 16, and a different set could surface different capabilities.
• Ranking. For ranking models by predicted performance on randomly held-out items, a plain benchmark average does slightly better than BenchMIRT. Its stated advantage is item-level detail, not better rankings.
• Misuse risk. The same discrimination estimates could be used to strip the most informative safety questions from a benchmark, producing an evaluation that an unsafe model could pass. The authors accept this risk in exchange for transparency.
• Subset figures are qualitative. The 10%/50% subsetting results are described only as "generally" or "often" matching the full set, so check exact agreement before using a reduced set in place of a full benchmark.