Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

What it claims

Game2Real-Bench pairs 1,000 procedurally generated toy-game questions with 400 curated real-world questions. The two sets are linked by the reasoning operation each question needs, not by visual similarity. Across 25 VLMs, model-level scores on the two sides correlate at 0.935 (both Pearson and Spearman). The authors add a point the summary omits: despite their simpler visuals, the game tasks are generally harder than the real-world ones and separate models better.

On transfer, the demonstrations were randomly selected one-shot game examples. They lowered fine-grained perception accuracy for all four models tested and slightly lowered average overall accuracy. Using game examples as naive demonstrations is therefore a net negative unless they are targeted at constraint reasoning.

How it works

The game side has 18 tasks under four capability groups:

• Fine-grained perception: HiddenObject, DotShape, MissingTiles, ColorSwap
• Spatial reasoning: CylinderColoring, CubeNet, RollingCube, BlockCounting
• Constraint reasoning: Hexakai, Pipe, Battleship, Tapa, BlockIsland
• Sequential planning: Laser, KeyPathQuest, GrowingTrail, RushExit, Mancala

Each task has a generator that samples a problem state and renders the image, question and answer together. Structural parameters (topology, constraints, paths) are kept separate from presentation parameters. A deterministic per-task solver re-derives each answer:

• Tasks that require a unique answer are checked for alternative solutions.
• Tasks that allow multiple answers keep all admissible ones.
• Tasks that require an optimal answer are checked against the solver's optimum.
• Every sample also passes shared checks for completeness, image validity and multi-image alignment.

Real-world samples come from existing benchmarks and datasets, including BLINK and IKEA Manuals at Work.

flowchart LR
  G[Game generators] --> V[Solver verification]
  V --> D[Dataset 23,893]
  V --> B1[Game eval 1,000<br/>disjoint seeds]
  S[Real sources] --> M{Matches reasoning need?}
  M -->|yes| R[Retain original QA]
  M -->|no| C[Reconstruct from media]
  R --> MR[Manual review]
  C --> MR
  MR --> B2[Real eval 400]
  B1 --> BENCH[Bench 1,400]
  B2 --> BENCH

For example, the authors aligned IKEA assembly instructions with assembly videos to build before/after-step questions. Samples with ambiguous answers or insufficient visual evidence were dropped.

Evidence

• Models: 25 total, of which 22 are open-weight (2B–35B total parameters; families include Qwen3-VL, Qwen3.5, Qwen3.8, InternVL3.5 and Gemma4) and 3 are proprietary frontier systems.
• Correlation: 0.935 Pearson and Spearman at the model level.
• Transfer: one-shot ICL on four models. Constraint reasoning improved for several models, fine-grained perception declined for all four, and the overall average declined slightly.
• Per-model scores: the fetched text ends at the experimental setup, so per-model scores and transfer deltas are not shown.

How to use it

Benchmark, dataset and code are at https://github.com/QingAnLiu/Game2Real-Bench. The page gives no install commands or licence.

• Dataset: 23,893 samples, of which 20,701 are open-ended and 3,192 single-choice, with 36,441 images including multi-image inputs. It is intended for training as well as evaluation.
• Contamination: the game eval set (600 open-ended, 400 single-choice) was generated with random seeds and layout configurations disjoint from the dataset. Overlap was then filtered with semantic fingerprints and image hashes. You can train on the dataset without obvious leakage into the game eval, though this is the authors' assertion.
• Executable generators and verifiers are released. You can generate more verified samples rather than relying on the fixed set.

Limits

• The real-world set is small: 400 samples across four groups, so roughly 100 per capability if evenly split. The page does not give the split.
• Transfer was tested only with one-shot, randomly chosen demonstrations on four models. Fine-tuning on Game2Real-Dataset is not evaluated in the fetched text.
• Tasks often involve several capabilities but are each assigned to one group, so per-capability results blur.
• The fetched page cuts off before the results tables and the thinking-mode and failure analyses (spatial representation, relational consistency, state updating). Check the full paper for those.
• No licence is stated on the page. Real-world samples inherit licences from about ten source datasets.