Many Eyes, One World: Feed-Forward 3D Reconstruction from Mixed Cameras

What it claims

MEOW is the publicly available MapAnything checkpoint, fine-tuned only on synthetic mixed-camera tuples. Its inputs are perspective, fisheye and full-360° panorama images, given as pixels only. On heterogeneous Stanford 2D3DS tuples it reaches 80.4 mAA@30, against 54.3 for Wid3R, which is given the camera type of every view. These figures come from the arXiv v1 abstract and introduction. The catalog summary's 79.9 vs 53.8 does not match this text. On the authors' laser-scanned mixed-camera benchmark, MEOW reaches 79.4 AUC@30 on mixed tuples against 29.3 for Wid3R. That gap is much wider than on 2D3DS, and the summary omits the baseline figure.

How it works

Inputs and outputs. The model takes N images and returns, for each view, per-pixel unit rays, depth along each ray and a camera-to-world pose. Poses are expressed in the frame of the first view, using the OpenCV convention. The model also returns one metric scale for the whole tuple. At inference the only auxiliary cues are each image's native shape (H, W) and a binary panorama flag, which the model infers from the pixels.

Input adaptations. These are small; the backbone is not redesigned:

• Every image is resized to a common tensor shape without cropping, so panoramas keep all longitudes.
• An aspect-ratio embedding restores the source layout information that resizing destroys.
• The dense head uses circular padding to handle the longitude seam of panoramas.
• The encoder is a shared DINOv2 feeding a multi-view transformer that alternates global and per-frame attention.

Losses. Solid-angle weighting accounts for each pixel covering a different amount of the sphere. A directional likelihood loss supervises the predicted rays.

LENSCOPE data engine. It uses a procedural indoor generator with 2,004 first-generation rooms and 499 denser, more realistic second-generation scenes. Scenes are built with CC0 textures, are reproducible from seeds and run on a single GPU. Each panorama pose is rendered once with analytic rays, radial depth and a validity mask, then resampled through one of seven camera models:

• OpenCV and pinhole (the majority)
• Fisheye624, EUCM and Mei
• Spherical crop
• Full panorama

Priors cover principal point, roll and tilt, and an optics layer adds blur, vignetting and noise. That layer is decoupled from geometry, so blur does not leak the field of view.

flowchart LR
  A[Seeded scene] --> B[Render panorama per pose]
  B --> C[Pose covisibility index]
  C --> D[Random walk: 2-8 poses, cov ≥ 0.40]
  D --> E[Resample each pose through sampled camera]
  E --> F{Views connected at cov > 0.25?}
  F -- yes --> G[Training tuple: RGB + exact rays, depth, poses]
  F -- no --> H[Widen FoV, square, untilt]
  H --> F
  H -. still fails .-> X[Discard]

Tuple sampling. A random walk over the offline covisibility matrix assembles 2–8 poses, adding a pose only when its covisibility with the current pose is at least 0.40. With probability 0.55 the cameras are "aimed": they converge on a shared point or look along a shared direction. Covisibility is then recomputed on the sampled views, because a narrow or tilted camera sees far less than its source panorama. A tuple is kept only if its graph is connected at a threshold of 0.25. A disconnected view is moved toward a wide, square, untilted camera and re-tested; if it still fails, the tuple is discarded.

Evidence

• 2D3DS mixed tuples: MEOW 80.4 mAA@30 vs Wid3R 54.3, with Wid3R given camera-type labels.
• Laser benchmark, mixed tuples, no fine-tuning for any method: MEOW 79.4 AUC@30 vs Wid3R 29.3.
• Ablations (stated qualitatively in the intro): synthetic adaptation substantially improves the backbone. Changing the input resizing alone does not. Full-FoV resizing and the aspect-ratio embedding both contribute.
• Scene realism: training on second-generation scenes instead of first-generation ones cuts Matterport3D zero-shot accuracy error by 17.4% and completeness error by 24.1%. If you build your own synthetic data, scene quality matters, not just camera diversity.

How to use it

The authors say the data engine, benchmark and evaluation pipeline "will be released." The page gives no release date, repository, licence or commands. It also never says whether MEOW's own weights will be released.

Limits

• Test images are resampled, not captured. The perspective and fisheye test views are resampled from real panoramas. Native fisheye optics, rolling shutter and real lens artefacts are not tested.
• Indoor only. All training scenes are procedural indoor rooms, so outdoor or large-scale generalisation is unshown.
• Panorama flag. The model relies on an inferred panorama flag and the native image shape, so misdetecting a panorama is a possible failure point.
• Incomplete page. The fetched text cuts off mid-method. Training compute, model size, full ablation tables and the authors' own limitations section are not visible.
• Version mismatch. The headline numbers differ between this v1 text and the catalog summary, so check which version you are citing.