F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation

What it claims

F2G-Pose regresses 6D pose, metric size and a 2,048-point completed camera-frame shape from a segmented RGB-D object in one forward pass. It does not use test-time sampling, candidate scoring or canonical-shape alignment. It needs no CAD model, reference images or explicit category label at inference. The authors (Fudan University and Shanghai Innovation Institute, October 2026) position it against diffusion and flow-based estimators: GenPose, GenPose++, Joint-RegDiff and RFMPose. They argue those methods couple accuracy to sampling and post-hoc candidate extraction under a latency budget. Against SecondPose, the closest direct foundation-feature method, the stated differences are dense shape supervision and MoE capacity across categories.

How it works

flowchart LR
  A[RGB crop] --> B[Frozen RADIOv2.5]
  B --> C[Project features onto points]
  D[Segmented point cloud] --> C
  C --> E[DGCNN tokens]
  E --> F[MoE Transformer, pooled]
  F --> G[Pose + size head]
  F --> H[Shape completion head]

1. Feature fusion. Frozen RADIOv2.5 layers 8, 16 and 23 are each compressed to 512 channels by 1x1 convolutions. Learned softmax weights mix them, and a 3x3 block refines the result to 1,024 channels.
2. Lifting. Each 3D point takes the feature at its projected 16-pixel patch index. Point and feature are concatenated, in the style of DenseFusion.
3. Tokenization. A DGCNN turns the fused points into 128 tokens of dimension 512.
4. Encoding. Each Transformer block uses an MoE feed-forward layer with 8 experts, top-2 routing per token and renormalized gate weights. A global pool follows.
5. Pose and size head. Rotation uses the continuous 6D representation with orthogonalization. Translation and size are regressed directly.
6. Shape head. An MLP predicts 512 coarse points. These are concatenated with farthest-point-sampled observed points, and a sigmoid confidence score selects 512 fused points. Each fused point then predicts 4 offsets, giving 2,048 points.

Only the task network is trained. RADIO stays frozen.

Losses:

• Pose and size: Smooth L1 on rotation, translation and size. The rotation loss is symmetry-aware: each axis is labelled as no symmetry, 90°, 180° or continuous symmetry. The loss is the minimum over the generated set of equivalent ground-truth rotations, taken in 6D space before orthogonalization.
• Shape: L1 Chamfer distance on both the fused and dense point sets.
• MoE balance: a Switch-Transformer-style load-balancing auxiliary loss.

Evidence

Training uses only pose-annotated synthetic data from 149 SOPE categories. Evaluation covers four settings:

• SOPE for synthetic accuracy.
• ROPE for synthetic-to-real transfer.
• HANDAL for additional real objects.
• HouseCat6D for camera-frame shape reconstruction.

The text reports state-of-the-art accuracy on SOPE and ROPE and 28 FPS for pose-only inference. The paper also includes analyses of latency, robustness to perturbation and occlusion, and GenPose++ candidate behaviour, plus pick-and-place robot trials. The fetched text ends partway through Section 3, so no metric values, baseline margins or GPU details are available here.

How to use it

The authors state that the code and pretrained models are on the project page, https://jrryzh.github.io/F2G-Pose/. The page gives no install commands or licence.

Inputs at inference are an RGB crop and a segmented partial point cloud. An upstream segmenter is therefore required.

Limits

• Segmentation dependency. Deployment is demonstrated only "under segmented RGB-D inputs".
• Fixed taxonomy. Generalization claims apply to the evaluated taxonomy (the 149 SOPE training categories), not to arbitrary categories.
• Speed. The 28 FPS figure is for pose-only inference. Shape output is described as "approximate" completed geometry.
• Missing details. The fetched page omits result tables, ablations, hardware and any limitations section. Check the full paper before relying on specific margins.
• Licence. Weights and code licence are not stated.