arxiv.orgpaper

MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning

MIGU is a modular framework for multimodal instruction grounding that fuses semantic priors from a vision-language model with a 3D geometric likelihood derived from eye-finger gesture geometry and uncertainty propagation (viewing-direction, depth, hand-direction error). The resulting Bayes-inspired grounding belief drives a behavior planner that either proceeds to manipulation planning or requests clarification, and is evaluated against baselines on a real-world mobile manipulation and tabletop task-and-motion planning benchmark. It matters as an example of explicitly modeling uncertainty across language and gesture modalities to make robot instruction-following more robust.