GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

What it claims

GameHorizon is positioned against three named gaps in prior gameplay datasets/benchmarks: they cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. The suite's fix is to pair a large instruction-annotated dataset with an evaluation design that has both a reproducible offline track and an online track used specifically to check that offline scores track real gameplay ability.

How it works

The offline track is the reproducibility layer: thousands of standardized questions organized into three primary tasks plus a series of diagnostic variants — this is finer structure than "offline track" alone conveys. The online track is not just a second score; its stated job is to test whether offline scores reflect actual gameplay capability and to localize failures to specific steps within long-horizon gameplay. GameHorizon-Data is described as the first large-scale AAA dataset with temporally aligned video, player actions, and multi-horizon instructions, built via the GameHorizon-Annotator pipeline (5,000 hours, 21 games, 100 expert players).

Evidence

The abstract reports only the scale of the study — 47 models, >1M invocations — yielding a "meaningful hierarchy of task difficulty" and "pronounced differences in model capabilities." No per-model numbers, no baselines, and no leaderboard appear on this page.

How to use it

The authors state they "will release our dataset, annotator, and benchmark." Treat this as a future release: no weights, no license, no download link, no run commands are given here. If you want the annotation pipeline or the offline question set, you would need to wait for or locate the actual artifact release.

Limits

This is an abstract-only page. The concrete task definitions, the metric per task, the actual difficulty ranking, which of the 47 models lead, and any license/usage terms are all absent — you cannot judge whether the benchmark is hard for a specific model class without the paper body or the release. The claimed novelty ("first large-scale AAA gameplay dataset") and the offline-online agreement claim are asserted, not shown, in the text available here.