FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity

What it claims

FlexPart, from Tencent Hunyuan with the University of Hong Kong and CUHK-Shenzhen (arXiv v1, 16 October 2026), segments a 3D mesh into the parts a text prompt lists. Coarse, medium or fine output comes from changing the prompt, not the model. Selective segmentation of named parts uses the same model, with everything unnamed assigned to a background label. With only part names from the user, it improves IoU by 35% over the strongest click-driven baseline on selective segmentation. With planner-completed prompts it beats the strongest baseline on PartObjaverse-Tiny by 15% mIoU, even when that baseline receives ground-truth guidance.

How it works

The training data comes from an automated Gemini pipeline. It takes artist-authored components from HY3D-Bench and merges them by semantic and functional relationships into three granularities. It then writes a control prompt for each level. This covers 57K objects and yields 150K prompt-target pairs.

Each prompt item is a part name plus a location clause and a shape clause, for example "right arm: upper right side of the torso, a tapering cylindrical limb." The i-th item maps to the i-th part ID. The authors report that this ordering prior gives more accurate assignments than permutation-invariant schemes.

Part IDs are 6-bit binary codes rather than RGB colours. RGB, as in earlier generative segmenters, produced fragmented predictions that needed colour clustering afterwards. The all-ones code is reserved for background, so the cap is 63 parts.

The model has two components:

• Part-aware VAE. It builds on LATTICE voxel-anchored latents. An 8-layer self-attention encoder reads a point cloud carrying position, normal and part ID into a 2048×64 latent. The decoder answers SDF queries on grid points and BCE-supervised part-ID queries at mesh face centres. Part indices are shuffled during VAE training so it cannot memorise order.
• Dual-stream DiT. It is in the style of MM-DiT and flow-based. Text tokens come from a frozen Qwen3-VL encoder with 1D RoPE. Latent tokens get 3D RoPE over voxel indices. A part-agnostic shape latent, produced by encoding the mesh with all IDs set to zero, is added to the noisy latent.

Training uses drop-to-background: parts are randomly moved out of the prompt into background items. The authors say this forces the model to follow the text rather than geometric priors. At inference a Qwen-VL planner looks at multi-view renderings and fills in the location and shape clauses. It can also expand a free-form command such as "segment the two swords and the hands" into target and background items.

flowchart LR
  A[User part names or command] --> P[Qwen-VL planner on renderings]
  A2[Hand-written full prompt] --> T[Qwen3-VL text tokens]
  P --> T
  M[Input mesh] --> S[VAE encoder, IDs zeroed]
  S --> D[Dual-stream DiT]
  T --> D
  D --> F[Part-ID field queried at face centres]

Labels are read at face centres, so the input geometry is never changed. Schema-driven methods instead synthesise separate part geometries.

Evidence

FlexPartBench is the authors' held-out benchmark. On it, FlexPart leads at each granularity and raises average mIoU by 22% over the strongest baseline. On PartObjaverse-Tiny, planner-completed prompts beat the strongest baseline by 15% mIoU. For selective segmentation, it gains 35% IoU over the strongest click-driven baseline. The fetched text does not name which baseline is strongest in each comparison, and it does not say whether the gains are relative or absolute.

Limits

The fetched text stops partway through the method section. Per-baseline tables, ablations, compute, and any stated limitations are not visible. The page gives no code, weights or licence, only a project page (zhao-yiqun.github.io/flexpart).

The 6-bit encoding caps output at 63 parts. Results with short user input depend on the Qwen-VL planner. FlexPartBench is the authors' own benchmark, drawn from the same annotation pipeline.