Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

What it claims

LensUnd gives an external 2D VLM a few well-chosen, target-marked views of a 3D-localized object, and its fine-grained descriptions can then be distilled into a 3D LLM that sees only its usual inputs. The two students are Video-3D LLM and 3DRS. The paper states the distilled students reach fine-grained captioning performance "comparable to a similarly sized 2D VLM" while keeping grounding and scene-level QA intact. The fetched text cuts off before the results tables, so no scores are available here.

How it works

LensUnd (Ground–Focus–Understand)

1. Ground. Any 3D grounder maps a ScanRefer-style query to a 3D box b. The grounder is used unmodified and only its predicted box is needed.
2. Focus. This step picks views of the target, using the "bbox-voxel" procedure: - Candidate frames must be geometrically valid and depth-visible. - Each candidate gets a quality score from projected target completeness, image coverage, centrality, and occlusion. Occlusion is estimated through depth consistency. - Each frame's depth pixels inside b are unprojected into a shared voxel grid, with about 12 cells along the box's shortest axis. This represents observed target surface, not projected image area. - Selection is greedy, as shown in the diagram below. Defaults are K = 4 views and a minimum temporal gap of δ = 12 frames.
3. Mark. The 12 edges of b are projected as a wireframe onto each chosen image. This lets the VLM tell which of several same-class objects is meant.
4. Understand. A 2D VLM gets these views plus a task instruction. Changing the instruction gives captioning, small-object grounding or QA. For small-object grounding, per-view 2D boxes are lifted to a 3D box using depth and camera parameters.

flowchart TD
  A[Candidates with quality + visible voxels] --> B{Any selected yet?}
  B -- no --> Q[Pick highest quality]
  B -- yes --> C{Any frame adds new voxels?}
  C -- yes --> G[Pick max new-voxel gain]
  C -- no --> Q
  Q --> U[Add to set, update covered voxels]
  G --> U
  U --> D{K frames or no feasible candidates?}
  D -- no --> B
  D -- yes --> E[Selected views]

LensDistill. This is asymmetric privileged-information distillation, applied only to fine-grained captioning.

• Teacher. LensUnd captions from the marked views produce y⋆, covering material, parts, state and spatial relations.
• Student. The 3D LLM receives 32 uniformly sampled native views with 3D positional encodings, the target center encoded as a <coord> token, and the task prompt.
• Training. Sequence-level cross-entropy, −log p(y⋆ | X, c(b), p).
• Inference. The LensUnd path is removed entirely, so there is no view selection and no VLM call.

Evidence

LensBench is a held-out set of 2,068 objects. Each object has three independently generated VLM descriptions, which a questionnaire-based audit supports as silver-standard references. The supplement (A.3, A.5) compares view coverage and caption quality across selection strategies.

Limits

• The fetched page ends mid-method, so the quantitative results, baselines and ablations were not visible.
• The page excerpt does not name the external 2D VLM.
• Distillation is only demonstrated for captioning. Small-object grounding and fine-grained QA are shown qualitatively (Figure 3) and only through LensUnd, which needs an external VLM at inference.
• LensBench references are VLM-generated rather than human-written, so scores measure agreement with another VLM's descriptions.
• LensUnd needs calibrated RGB-D with camera poses, and the setting is indoor scenes.
• The student still needs a target center at inference, so it depends on an upstream grounder.
• The excerpt mentions no code, weights or licence.