Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

What it claims

Imagine3D-LLM is a multi-view MLLM with an auxiliary 3D Gaussian Splatting head. It is trained to reconstruct a coarse scene before it answers. The fetched text reports gains on seven spatial reasoning and 3D scene understanding benchmarks, but the excerpt ends before any results tables, so no figures can be given here. The paper is a NeurIPS 2026 paper from KAIST AI, ETH Zürich, Google and TUM. The project page is https://cvlab-kaist.github.io/Imagine3D-LLM.

How it works

The summary leaves out several details that matter for anyone reimplementing this:

• Token placement. M learnable Gaussian summary tokens sit between the image tokens and the text tokens, with M much smaller than the image token count. Because attention is causal, the summary tokens see all the visual evidence and the text tokens see the summary tokens.
• Why separate tokens. The authors predict Gaussians from dedicated tokens rather than from the image features. They say this stops the model from copy-pasting 2D content into image-aligned Gaussians. The small M is a deliberate bottleneck that forces content seen in several views to merge, and the authors report this produces emergent object-centric grouping.
• Decode layer. Hidden states are read from a middle LLM layer ℓ, not the final layer. This leaves later layers free for text tokens to attend to the scene summary. The paper says Tab. 5 validates this choice; its values are not in the excerpt.
• Decoder. A lightweight MLP head maps each token to G Gaussians, giving M×G Gaussians per scene without lengthening the sequence. Each Gaussian has a center, opacity, covariance and spherical-harmonic color coefficients.
• Losses. The losses are language modeling, a reconstruction loss on renders at the input viewpoints (weighted MSE plus LPIPS), and a distillation loss from a pretrained compact Gaussian teacher, ZipSplat. The summary omits this last loss.
• Why distillation. The authors found that pure photometric supervision through the LLM converges too slowly to be practical for MLLM finetuning. The teacher's token count M is set equal to the student's so that tokens align one-to-one.

flowchart LR
  I[Multi-view images] --> LLM
  LLM -->|middle layer ℓ| GS[Gaussian summary tokens]
  GS --> H[MLP head: G Gaussians/token]
  H --> R[Rasterize at input views]
  R --> Lr[MSE + LPIPS loss]
  T[ZipSplat teacher, M tokens] --> Ld[Distillation loss]
  H --> Ld
  LLM --> Llm[LM loss]
  Lr --> L[Total loss]
  Ld --> L
  Llm --> L

Evidence

The excerpt contains only qualitative analysis. With the reconstruction objective, image-to-image attention between corresponding features gets sharper. PCA of the image features also shows the same object encoded consistently across views, compared with the baseline. No benchmark numbers, baselines or compute figures appear in the fetched text.

Limits

• The fetched text stops partway through the method section. Results, ablations and training cost are missing, so check the full PDF before relying on the gains.
• The base MLLM, the values of M, G and ℓ, and the loss weights are not given in the excerpt.
• ZipSplat encodes its inputs with VGGT, a 3D geometry foundation model. Training therefore still depends on a 3D foundation model indirectly, even though the method is framed against fusing such features. The excerpt does not say whether the teacher is needed at inference.
• Figure 1's caption cites the teacher as [1] (C3G), while the text cites ZipSplat as [72]. Confirm which model is used.
• The excerpt does not mention code or weights.