Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

What it claims

Plan-Bench scores VLMs on embodied reasoning and planning from images alone, with no simulator or controller involved. It has three dimensions:

• Spatial Cognition: 259 items, 6 subtasks.
• Temporal Reasoning: 210 items, 5 subtasks.
• Multi-Step Planning: 230 items, 5 subtasks.

Of the 699 items, 516 are open-ended and 183 are single-choice. The best of 28 models, Qwen3.8-Max, reaches 75.0% overall. A random baseline scores 6.5%.

How it works

• Task types. Subtasks are concrete:
• Occlusion Order (remove repeated objects across views in order)
• Mirror Distinction (tell reflections from real objects)
• Persistent Counting (count across frames without double-counting)
• Progress Tracking (match frames to a step list and ignore irrelevant actions)
• Dependency Planning (retrieval order from support relations)
• Capacity Planning, Route Planning (shortest route from a floor plan), Rearrangement Planning, and others

Answers are typically option letters or sequences, such as an image ordering or an object-removal order.

• Image sources. Subclasses that need controlled layout or viewpoint are rendered in Three.js or OmniGibson. Subclasses that benefit from natural images draw on existing datasets, including CLEVR and CaptainCook4D. Temporal tasks are largely egocentric multi-image sequences, such as cooking and navigation.
• Curation. The authors hand-wrote over 2,500 QA pairs. Each was then reviewed by three annotators who had not written it, and kept only if all three agreed on correctness, image-QA consistency and task relevance. That left 699 items, about 28%.
• Scoring. DeepSeek-V4-Flash acts as an LLM judge. It sees the model output, the ground truth and an "answer hint" listing valid alternatives, such as permitted order swaps or alternative object names. On 200 randomly sampled items the judge agreed with human evaluation 100% of the time. The judge prompt is in Appendix B.2.

Evidence

The visible part of Table 1 gives accuracy (%) per dimension:

The random baseline counts open-ended items as 0% and single-choice items as 25%.

The two leading models are close on temporal reasoning but 18 points apart on planning. Spatial cognition is the lowest dimension for Qwen3.8-Max.

The 28 models are 23 open-source and 5 commercial. They are grouped by size: 2–4B, 7–12B, 27–31B dense or MoE, and commercial frontier models. The families include Qwen3.8, Qwen3.5, Qwen3-VL, Gemma-4, Step3-VL, GLM-4.6V, InternVL3.5, Ovis, SenseNova-U1, NEO and MiMo-VL. The authors report that accuracy generally rises with scale, with variation across families.

How to use it

The benchmark and evaluation code are at https://github.com/MinneFeng/Plan-Bench. The page gives no install commands, licence or judge-cost figures. Running it as published requires access to DeepSeek-V4-Flash as the judge.

Limits

• Truncated page. The fetched text cuts off partway through Table 1. Rows for the other three commercial models and all 23 open-source models are missing, as are the ablations behind the "larger models as planners" claim. Check the full paper before comparing a specific model or size class.
• Small sample sizes. 699 items across 16 subclasses averages about 44 per subtask. Per-subtask scores will be noisy.
• Plans are not executed. Plans are judged as text answers against a reference. A high score shows a correct answer, not that the plan works on a robot.
• Narrow judge validation. Correctness on 74% of items rests on one LLM judge plus hand-written hints. The 100% agreement figure comes from a single 200-item check.
• Inherited licences. Some images come from existing datasets, so their licences may apply. The page does not address this.