DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

What it claims

DK-VLM adds a geometry branch to a pretrained Qwen-VL decoder without replacing its native mRoPE attention. It is built on two backbones, Qwen2.5-VL-7B and Qwen3-VL-8B, and the trainable parameters come to 3.86% of the backbone's parameter count. The 3D points and camera poses are estimated from the input images, so inference needs no ground-truth depth or poses. Evaluation covers seven spatial benchmarks plus VideoMME and MVBench for general video. The authors also report that RC supervision helps DK-VLM more than it helps a Qwen baseline fine-tuned on the same instruction data, which they take as evidence that the two components complement each other.

How it works

flowchart LR
  H[Layer hidden states] --> S[Semantic kernel: native Q/K/V + mRoPE, causal]
  H --> G[Geometric kernel: new Q/K/V, full attention]
  F[Geometry frontend: patch XYZ + camera pose] --> G
  S --> M{Per-layer scalar gate}
  G --> M
  M --> R[Residual stream]
  G -.-> L[L_pair loss]

• Selected layers only. Dual-Kernel Attention replaces attention at selected decoder layers. All other layers run the native computation. The page does not say which layers are selected.
• Geometric kernel.
• Its Q/K/V projections start as copies of the native ones and are then trained separately.
• It operates only on visual tokens that have valid geometry. Text tokens are excluded.
• It uses full, non-causal attention, so that earlier frames can use geometric evidence from later frames.
• GTA split. Each geometric head is split into three channel groups:
• Point GTA: a 4×4 SE(3) transform per 4-channel block, with the patch's 3D point as translation scaled by a per-block λ, and the source-view rotation.
• Camera GTA: the frame's camera-to-world pose, expressed relative to the first frame.
• Remaining channels: left untransformed.
• Relative geometry in the attention. The transforms are applied to Q, K and V before attention, and the inverse transform is applied to the output. As a result, both the attention scores and the value aggregation see relative transforms.
• Fusion. The two kernels share one output projection, initialized from the native projection. A scalar gate per layer mixes them: Y = (1−α)·φ(S) + α·φ(G). The gate is initialized to α = 0.5. Positions outside the geometry-valid set use the native output projection.
• Swappable frontend. The model consumes geometric states (points and poses), not frontend-specific latent features. The authors therefore say any compatible geometry frontend can be substituted. They cite several candidates but do not name the one they used in the text shown.
• Training. One stage that jointly optimizes the instruction loss and a pairwise geometric loss (L_pair). L_pair supervises the geometric kernel's output before fusion.
• RC chains. Verified single-step relations are first normalized to the direction convention of their source. Relations that share landmarks or views are then linked into paths. The heading at each step becomes the reference frame for the next, and every step's target (e.g. "right", "left", "straight") is recomputed from the source annotations.

Evidence

The text available states only the headline numbers: 74.33 on VSI-Bench, 58.28 on ReVSI-32, and the best six-benchmark average among the methods compared. It also says RC brings "consistent gains" on route-planning and viewpoint tasks. No baseline scores, deltas or per-benchmark tables appear in the fetched portion.

Limits

• The fetched text stops partway through the method section (RC stepwise supervision). Experiments, tables, ablations and appendices are missing, so baselines and margins cannot be checked here.
• The page does not mention code, weights or a licence. Only a project page is linked: https://sumuru789.github.io/DK-VLM-website/.
• Performance depends on the quality of the estimated geometry. Inference requires running a separate geometry frontend, and its cost is not stated.
• RC targets are only as good as the "verified" single-step annotations behind them. The verification procedure is described only in Appendix A.4, which is not in the fetched text.