GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

What it claims

GeoVerse runs multi-view diffusion inside the frozen feature space of Depth Anything 3 (DA3), building on Geometric Latent Diffusion (GLD). It conditions that diffusion on intermediate features from a frozen Wan2.2 VACE video model and on reprojections from a growing colored point-cloud memory. Two practical changes over GLD stand out:

• Training data: the corpus grows from 4 to 15 real and synthetic multi-view datasets.
• Inference speed: reflow distillation cuts denoising to 4 steps, taking per-inference latency from over 100 s to under 10 s.

Unlike video-based NVS pipelines, it generates target cameras directly rather than sampling dense in-between frames.

How it works

flowchart LR
  C[Context RGB] --> M[Point-cloud memory]
  M -->|reproject| G[Target RGB-D hints + validity mask]
  C --> DA3[Frozen DA3 encoder to Level 1]
  G --> DA3
  G --> W[Frozen Wan2.2 VACE, one forward pass]
  W -->|ControlNet adapter| D[4-step flow denoiser]
  DA3 --> D
  D --> H[RGB + geometry heads]
  H -->|scale-aligned depth| M

• Input. GLD zero-pads target slots. GeoVerse instead fills them with memory reprojections, masked by validity, before encoding with frozen DA3. DA3 features are taken at blocks 5/7/9/11, and Level 1 is the synthesis boundary.
• Denoising. Flow matching runs on Level-1 features. A conditional cascade then samples Level 0. The remaining frozen DA3 blocks compute Levels 2 and 3. RGB and geometry heads decode images, depth and raymaps.
• Video prior. Wan2.2 VACE is not sampled. Its features are extracted once and reused at every denoising step:
• Noise with σ = 0.1 is added only to valid target projections.
• Features come from DiT blocks 0/5/10/15.
• A ControlNet-style branch with zero-initialized projections adds these features as residuals.
• Loss. The flow-matching loss weights context views 0.25 and target views 1. It is combined with an ℓ1 RGB loss, a high-gradient ℓ1 loss, and a scale-aligned depth ℓ1 loss.
• Memory. The memory is initialized from DA3-predicted depth, or metric depth if available. Each round's predicted depth is scale-aligned and fused back into the memory. This is the loop that supports long trajectories.

Evidence

On DL3DV (Table 1, excerpt):

Notes on the table:

• GLD has the best baseline ATE (0.058), so it is the strongest geometric competitor.
• CAMEO is the fastest baseline at 14.78 s, but it is far worse on every quality metric (PSNR 11.14, ATE 0.906).
• NeoVerse posts the lowest Reproj. error of the baselines shown (0.638).

Limits

• Truncated text. The fetched page cuts off mid-table. GeoVerse's DL3DV ATE, its other 3D metrics and its measured time are not readable. The Mip-NeRF360 result (32.4% lower ATE vs GLD) appears only in the abstract, and no ablations are visible.
• Latency claim. The "under 10 s" figure comes from the introduction, with no hardware stated. The heavy baseline runtimes are not directly comparable without that information.
• Two frozen dependencies. Inference needs both DA3 and Wan2.2 VACE.
• Release status. No code, weights or licence are mentioned beyond the project page, geoverse-nvs.github.io.
• Error accumulation is untested here. The memory relies on scale-aligned predicted depth. Errors that compound over many rounds are not addressed in the visible text.