SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

What it claims

SpatialChain gives each spatial GQA question a reference reasoning chain derived from the scene graph. A model's thinking trace can then be audited for whether it engages the correct spatial relations, not only whether its final answer is right. On the 895-item binary test set, LoRA-fine-tuning Qwen3-VL-8B on the 28,350 training chains gives the top accuracy, 85.03% against 78.88% for its base. GPT-5.4 is the weakest model at 72.18%, trailing even base Qwen3-VL-4B (78.44%).

How it works

Every retained chain was generated with access to the scene graph and was kept only if it reached the correct answer:

flowchart TD
  A[GQA balanced train/val] --> B{17 relation-centric types?}
  B -- no --> X[dropped]
  B -- yes --> C{GPT-5-nano: needs spatial inference?}
  C -- no --> X
  C -- yes --> D[Sample 30k train / 900 test]
  D --> E[Claude Haiku 4.5 extended thinking<br/>given scene-graph JSON]
  E --> F{Answer matches GQA ground truth?}
  F -- no --> X
  F -- yes --> G["Stored as think / answer pair"]

• Test-set selection. The test set keeps the 1,000 candidates with the longest GQA semanticStr, which biases it toward multi-hop questions.
• Chain generation. Haiku ran with a 3,000-token thinking budget and a 4,096 max output, via the Message Batches API. It was told to write as if looking at the image and never to cite scene-graph fields.
• Unused candidates. 56,773 filtered candidates were never given chains and are unused.
• Judge. Claude Sonnet 4.6 receives the question, the ground truth, the scene graph, and the candidate's answer and reasoning. It scores answer correctness, reasoning faithfulness and reasoning completeness, each in [0,1].
• Overlap metrics. ROUGE-1/L, token F1, entity overlap and length ratio are reported. The authors state that no faithfulness conclusion rests on these.

Evidence

Results on the 895-item binary test set. Unparsed outputs are excluded from accuracy: InternVL3.5-4B 25, InternVL3.5-8B 16, Sonnet 4.6 7.

• Fine-tuning shifts answer polarity. It raises Yes-accuracy by about 12 pp but lowers No-accuracy by about 4 pp. The test set's yes-rate is 0.63, and the paper treats positive-answer bias as a marker of language-prior use. GLM-4.1-9B is the only model with balanced polarity.
• Judge validation. On 198 human-annotated items, judge–human agreement matches human–human agreement. A second judge, GPT-5.6, preserves the model ranking (ρ = 0.88).

How to use it

• Code and data. Data, generation scripts and evaluation code are at github.com/spatialchain/SpatialChainBenchmark. No licence is stated on the page.
• Fine-tuning recipe.
• LoRA with r=16, α=32 and RSLoRA, applied to all linear layers.
• 2 epochs at an effective batch size of 12, on one A100 80GB.
• The adapter is merged into the base weights before evaluation.
• Inference settings.
• Open models run in vLLM with τ=0.6, top-p=0.95, top-k=20 and a 16,384-token maximum.
• InternVL3.5 needs an extra prefix to enter thinking mode.
• Proprietary models are run at their default temperature.

Limits

• Only conclusions are verified. The answer filter checks the final answer, not each step. A reference chain can contain a locally unsupported step.
• Relation skew. left_of/right_of account for 62.7% of training questions, labelled by the first relation in each question. Per-relation test cells with n<20 are non-inferential.
• Overlap metrics favour Haiku's style. The references were written by Claude Haiku, so overlap rewards Haiku-like prose. Fine-tuned models inherit that style, which inflates their R-1/F1 of about 0.65.
• Judge overlap with evaluated models. The judge, Claude Sonnet 4.6, is also one of the evaluated models.
• Possible out-of-distribution regression. The authors flag a possible "stylistic specialization" effect on external spatial benchmarks after fine-tuning. They propose replay-augmented training as a mitigation; the fetched text does not show it tested.
• Small, binary test set. The test set has 899 items, effectively all yes/no.
• Truncated page. The fetched text cuts off during the results section. The per-model shortcut rates, the judge scores and the external-benchmark numbers were not visible.