InfiniHand: Streaming World-Space Hand Motion Estimation from Egocentric Video

What it claims

InfiniHand predicts per-frame camera intrinsics and pose, plus MANO pose, shape, global orientation and root translation for each hand, in world coordinates, from uncalibrated egocentric RGB. It is not a new backbone. It is built on the LingBot-Map streaming 3D foundation model, with a WiLoR hand encoder bolted on. "Feed-forward" is slightly loose: camera poses are refined by a sparse keyframe bundle adjustment using a DROID backend after prediction. HaWoR, by contrast, runs a full DROID-SLAM plus Metric3D cascade.

How it works

Stage I (camera-space).

• Hand masks. A DPT decoder reads four layers of LingBot-Map features and predicts left and right hand masks. The decoder is initialized from LingBot-Map's pretrained depth head, with the final layer swapped for a 2-channel mask head. It is trained with BCE plus Dice loss.
• Boxes. Masks are thresholded at confidence > 0.9 and the boxes are expanded 1.5×.
• Feature fusion. For each box, cropped backbone patch features are concatenated with WiLoR-encoder features from the RGB crop and fused by a 1×1 conv.
• Hand parameters. A MANO head regresses pose, shape, orientation and log root depth. An auxiliary head predicts 2D joints.
• Lateral translation. x/y translation is recovered by differentiable least-squares projection against the 2D joints with depth held fixed. This step is borrowed from ViDiHand.

Stage II (world-space).

• Joint training. The hand modules are trained jointly with LingBot-Map's camera head.
• Streaming memory (GCA). The memory holds 4 anchor frames, a dense window of the 16 most recent frames, and compressed trajectory tokens that survive after dense features are evicted.
• Training clips. Clips use a 4 + 16×2 frame layout, and memory is reset between clips.
• Box noise. Ground-truth and predicted boxes are mixed 1:1 so the hand head learns to tolerate localization noise.
• World transform. Camera-space hands are mapped to world coordinates with the BA-refined trajectory.

flowchart LR
  V[Egocentric frames] --> B[LingBot-Map + GCA memory]
  B --> M[Mask head → boxes]
  B --> C[Camera head]
  M --> F[Fuse backbone crop + WiLoR crop]
  V --> F
  F --> H[MANO + LSP translation]
  C --> BA[Keyframe sparse BA]
  H --> W[World-space hands]
  BA --> W

Training data

The corpus is about 5,000 hours from nine public datasets, sampled at 10 FPS:

• ARCTIC, HOT3D, EgoDex, DexYCB, HO3D and H2O-3D
• EgoVerse, EgoLive and Xperience-10M

Datasets with only 3D keypoints were MANO-fitted. Masks and boxes come from rendering the MANO meshes. Labels were cleaned against 2D keypoints from Sapiens 2, using the 90th-percentile box-normalized joint error per sequence. This filtering removed about 30–40% of candidate data, with EgoDex called out as the noisiest source. Some borderline sequences were inspected manually. Stage II uses only the subset that has ground-truth camera trajectories.

Evidence

The fetched text cuts off in the middle of the method section, so no results tables, ablations or experimental settings are available. The only figures come from the abstract:

• 21.4% lower ARCTIC PA-p than ViDiHand.
• 11.19 FPS, more than 2× HaWoR's throughput. The hardware is not stated.
• "Substantially mitigated" world-space drift, with no number given.

Limits

• Truncated page. The ablations are not visible, for example the WiLoR fusion, the box mixing and the effect of BA. Neither are drift metrics or any limitations section. Read the full paper before relying on the gains.
• Dependencies. Reproduction depends on LingBot-Map weights, WiLoR weights and a DROID BA backend. The page gives no code, weight release or licence, only the project page https://infinihand.github.io/.
• Data licences. The training corpus inherits the licences of nine datasets, and reproducing it requires both MANO fitting and the Sapiens-2 cleaning pipeline.
• Scale and evaluation risk. Metric scale relies on the 10–16 frame memory windows and the camera head. In-domain benchmarks such as ARCTIC are also part of the training corpus.