StudentBench: AI and human tutoring yield equivalent GRE learning gains

What it claims

StudentBench is a public evaluation platform plus two studies testing whether LLM tutoring matches expert human tutoring on GRE learning gains. The headline result the summary states (statistical equivalence, ~918x cheaper) rests on specifics the summary omits: the equivalence test returns p = .015 overall, and the single AI tutor that matched humans at 918x lower cost did so at p = .044, costing USD 0.0052 per percentage-point gained versus USD 4.81 for the human tutor. In five of seven GRE domains, the best-performing AI tutor surpassed the human tutor on average.

Evidence

• Learning-gains study: 2,383 participants split across AI tutoring, human tutoring, and no tutoring, on Quantitative and Verbal GRE questions. Over 175,000 student-AI messages collected via the platform.
• Rubric study: 2,028 pairwise evaluations by expert human tutors comparing LLM-generated lesson plans and practice problems.
• Together the studies separate AI tutors on five axes: lesson planning, practice-problem creation, conversational pedagogy, cost, and engagement.
• Mechanism finding (Quantitative sessions only): a correlation chain where faster AI reply latency → more student messages → more correct practice → larger learning gains, each link at p < .002. This is the one actionable design signal in the abstract — reply speed, not just answer quality, appears to drive the funnel.

How to use it

The StudentBench platform is stated to be freely available (linked from the paper). The abstract gives no model names, no licence, and no API details.

Limits

This is an abstract-only arXiv page (v1, submitted 23 Sep 2026); the numbers above are all it establishes. Not stated: which LLMs were used as tutors, how "expert human tutor" was defined or recruited, the equivalence-margin used for the p-values, participant demographics, or whether the cost figure includes any human-in-the-loop overhead. The mechanism chain is reported for Quantitative GRE only — no claim is made for Verbal. Results are GRE-specific; generalization to other subjects or longer-horizon learning is untested per the text. Verify the platform's licence and the underlying models before building on it.