arxiv.orgpaper

Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation

This arXiv paper proposes augmenting generated robot manipulation videos with synthesized contact audio, using the loudness of generated sounds to derive a time-varying desired-force profile for contact-rich tasks. The pipeline drives a Franka Panda robot via a closed-loop force regulator tracking the audio-shaped force curve, outperforming kinematic-only video-generation baselines, and doubles as a data-generation engine for training closed-loop manipulation policies.