ReviewBench: An open benchmark for AI code review

What it claims

ReviewBench predicted the direction of a production A/B test for Copilot code review (CCR) on every metric the page reports. The test was a "lite-tier" change: a multi-model ensemble that merges several independent model runs into one review, replacing a single run.

• Offline prediction: ReviewBench predicted higher precision, higher recall, more comments, and lower cost per review.
• Online result, relative to the production control:
• Addressed rate (the precision proxy) rose 8.0%.
• Recall rose 13.6%.
• Comment volume rose 61%.
• Cost per review fell 8.0%.
• Severity shift: ReviewBench predicted a 227% increase in critical comments; production showed 262%. Both showed the same move toward more moderate comments and fewer nits.

The authors say this matches a broader pattern. Across experiments run on ReviewBench before A/B testing, the offline direction has consistently matched production. Note the claim is about direction, not magnitude.

How it works

The details below are not in the summary.

• Pull-request size is deliberately not representative. Language and repo-size distributions mirror GitHub. PR size is reweighted toward "the reviewable middle and tail" to reduce tiny single-file changes in favour of multi-file PRs. Absolute scores will therefore not match what you would see on a typical PR stream.
• Which recall to compare on. Augmented recall expands the denominator based on what each agent discovers, so its values are not comparable across systems. The authors use grounded recall as the headline cross-system metric and treat augmented metrics as a per-system diagnostic only.
• Re-ranking by preference. Changing β in the Fβ score, or slicing by severity or category (for example, security- or privacy-focused review), re-ranks the leaderboard.
• Versioning. The dataset, the judge and the matcher are versioned for every evaluation. Results are only comparable under the same benchmark configuration and are revalidated when it changes.
• Online proxies used for the A/B test:
• Addressed rate: the percentage of CCR comments that an LLM judges prompted a corresponding code change. The judgement uses the diff, thread, reactions, resolution state and post-review code.
• Online recall: measured as how much additional human review was still needed.

How to use it

• ReviewBench is a research preview, available through the ReviewBench website. There you can browse the benchmark, compare agents and "bring your own agent" to evaluate and submit results.
• The full dataset is stated to be public. The 187 source repositories are open-source licensed.
• The fetched page gives no install command, API shape, submission format or licence for the benchmark itself. Check the site for these.

Limits

• Single worked example. The evidence for production alignment is one ensemble experiment plus an unquantified claim of consistent directional agreement. No count of experiments or failure cases is given.
• Self-evaluation. The validation is GitHub evaluating its own product, and the online precision proxy is itself LLM-judged.
• Missing details. The fetched text names the three-stage golden-set process and the six metrics but does not list the stages or the metric definitions. They are not reproducible from this page.
• The 96.6% agreement:
• It comes from an internal audit by senior engineers who did not build the dataset, not an external review.
• It measures true/false-positive judgements on golden-set findings.
• Validity threats. The authors say they publish known threats to validity separately. These are not in the fetched text.
• Status. The release is labelled a research preview, so expect version changes that break comparability with earlier results.