HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

What it claims

HARMONY reconstructs a complete, compositional 3D indoor scene from a single monocular image, aiming for both accurate inter-object spatial relationships and high-fidelity geometry — two things the paper frames as a tradeoff in prior work. Agentic reasoning gives good semantic/spatial understanding but poor pixel alignment; visual geometry foundation models give dense point maps but limited reconstruction quality. HARMONY combines both in a hierarchical chain-of-thought framework and reports outperforming the evaluated reconstruction baselines, with qualitative comparisons against GPT-6 Astra showing more faithful object arrangements and better detail preservation.

How it works

Starting from an empty 3D floorplan, the pipeline runs in stages:

1. Camera calibration against the reference image to establish a semantically-grounded spatial frame.
2. Agentic VLM reasoning to recover the 3D room layout and an initial object placement order.
3. Hierarchical placement in dependency order: wall-mounted elements → free-standing furniture → dependent decorations sitting on furniture. Furniture uses depth-first traversal so each placement is conditioned on already-resolved structure.
4. Geometry-based refinement after each VLM placement, using point-cloud estimations so the rendered image aligns better with the input.

A reflective feedback loop across placements is used to curb error accumulation. The key design choice is interleaving the VLM's semantic placement with per-object geometric refinement, rather than doing all reasoning then all geometry.

flowchart TD
  A[Monocular image] --> B[Camera calibration]
  B --> C[VLM: room layout + placement order]
  C --> D{Placement queue by hierarchy}
  D --> E[Place object via VLM]
  E --> F[Point-cloud geometric refinement]
  F --> G{Aligned?}
  G -- no --> H[Reflective feedback] --> E
  G -- yes --> D
  D -- empty --> I[Composed 3D scene]

Evidence

The abstract states experiments on both synthetic and real-world images, and claims HARMONY outperforms the evaluated reconstruction baselines. No metric names, numbers, dataset sizes, or baseline identities are given in the fetched text.

Limits

This is an abstract only — no quantitative results, no ablations, no code or weights are stated on this page. The comparison to GPT-6 Astra is described as qualitative, not measured. Scope is explicitly indoor scenes; outdoor or non-room imagery is not addressed. The method assumes objects decompose into the wall/furniture/decoration hierarchy, so scenes that violate that dependency structure are untested. Reliance on both a VLM and a geometry foundation model implies non-trivial compute per scene, though the page gives no cost figures. Verify the actual metrics, baselines, and any code release in the PDF before adopting.