Reliability-Gated Fusion of Consumer Head and Foot IMUs for Lower-Body 3D Pose

What it claims

On a consumer head-plus-feet IMU layout (AirPods head IMU, two Striv smart insoles, no pelvis sensor, no foot gyroscope), the reliability problem is per channel, not per sensor. In the channel ablation, foot acceleration alone is the best input configuration tested: 66.6 mm, against 79.0 mm head-only. Foot orientation alone is worse than no feet (p=0.054, two-sided). Adding the foot orientation channels to foot acceleration wipes out most of the gain (Hodges–Lehmann +12.9 mm, p<10⁻⁵). The insole firmware's fused Euler orientation is biased, and with no raw gyro that bias cannot be filtered out.

The proposed fix gates each channel block (orientation, acceleration) of each stream separately, so the model can keep a stream's acceleration while shutting off its orientation. The authors state they claim no new gating mechanism. The contribution is applying and validating it for test-time sensor faults.

How it works

The backbone is an IMUPoser-style recurrent model:

• Input: each sensor gives a 12-D vector (3 acceleration + 9 orientation-matrix values), 36-D in total. Acceleration is scaled by 30 m/s².
• Stems: each stream's stem is split into acceleration 3→64 and orientation 9→64.
• Gates: a shared Conv1d (kernel 9, ±133 ms, non-causal by default) → ReLU → sigmoid emits one scalar gate per stream×block. Each gate scales its own 64-D half.
• Fusion and trunk: the gated features are concatenated (384-D), passed through Linear→512 with ReLU, then a 2-layer BiLSTM (512 per direction, input dropout 0.2).
• Outputs: 6D rotations for 9 lower-body joints, forward kinematics with fixed SMPL bone lengths, and a contact head.

flowchart LR
  A[AMASS synthetic IMU<br/>+ foot corruption p=0.5] --> P[Stage 1 pretrain<br/>pose loss + gate BCE]
  P --> F[Stage 2 fine-tune<br/>real pseudo-GT]
  F -->|gate BCE off,<br/>corruption kept on| M[Gated model]
  F -.->|gate BCE kept on| X[Gates forced open<br/>against task loss]

Training runs in two stages:

1. Pretraining uses AMASS (CMU, BioMotionLab NTroje, MPI HDM05) with virtual sensors at the head and both feet, and no synthesized foot gyro. Foot streams are corrupted per window with probability 0.5 by constant bias, slow drift, or dropout bursts. - Gates get an auxiliary BCE loss with weight 0.1. - Bias and drift mark only the orientation block as unreliable. Dropout marks both blocks. - Settings: ℓ1 on 6D rotations, Adam lr 3e-4, 40 epochs, reaching 6.9° geodesic error on held-out synthetic data.
2. Fine-tuning uses an aligned-FK ℓ1 position loss under per-window similarity alignment. - Settings: 5 s windows (150 frames), batch 32, lr 1e-4, 20 epochs, final checkpoint. - Gate supervision is turned off, because the real "clean" foot streams are natively biased and keeping the BCE forces the gates open. - Corruption augmentation stays on so the gates keep their selectivity.

Evidence

Channel-gated results (3 seeds) for clean / bias / drift / dropout: 69.4 / 77.8 / 69.6 / 72.8 mm. It is the best of the learned fusion variants in every regime and sign-consistent in 7/7 classes. On clean data, static gates score 83.7 mm and ungated 86.6 mm.

Supporting findings:

• Gate selectivity comes from the auxiliary supervision. A control trained without the auxiliary loss leaves its gates high and unselective.
• Stream-level gates add little over static weights. The authors report this as a negative result.
• Dropping a known-bad channel in advance is brittle. It holds up under foot faults but collapses when an unanticipated stream fails: under head dropout it scores 92.9 mm against 79.3 mm for the gated model.

HMD-Poser comparison (single seed): the fine-tuned HMD-Poser is more accurate on clean data (64.4 mm) and nominally under drift. There is no significant paired difference under bias or dropout. The gated model runs at 4.6× the CPU throughput with 37% fewer parameters.

Dataset: 35 takes (5 runs × 7 motion scripts), 82,417 frames (45.8 min at 30 FPS).

Label-integrity check: re-scoring head-only predictions against 4-view fused labels instead of single-view labels changes error by only 0.6 mm (83.9 vs 84.5).

Correction of the earlier workshop pilot: its finding that adding feet hurts shrinks to non-significant on the corrected pipeline (80.3 vs 79.0 mm, p=0.41).

How to use it

Code is at https://github.com/ZhilinGuo/reliability-gated-imu-fusion. The page excerpt gives no install commands or licence.

Requirements stated on the page:

• Hardware: the setup assumes AirPods (streamed through a custom iOS app) and Striv insoles over BLE at about 30 Hz.
• Cameras at training time: 4 Intel RealSense cameras are needed to produce labels.
• Calibration at deployment: per-take mounting calibration still depends on vision.
• Alignment: there is no hardware sync. Streams are aligned after capture by cross-correlating motion against SAM 3D Body output.

The release includes the capture, alignment and pseudo-GT protocols.

Limits

• One subject only, who is an author.
• Labels are SAM 3D Body pseudo-GT, not mocap. The authors say these labels are noisier under occlusion, fast motion and turns. The label check validates the choice of camera view, not absolute correctness.
• Sensor-to-bone calibration is transductive. It is fitted per take against that take's own labels.
• Faults are simulated. Drift and dropout appear only at evaluation; bias also appears in training.
• Default gates are non-causal. A fully causal variant is evaluated in Sec. 6.4.
• The HMD-Poser comparison is a single seed.
• No pelvis sensor, so root translation and hip axial rotation are weakly observed.
• Reported negative results: one-time T-pose calibration, oracle hard masking, and augmentation without gating.
• The fetched text is truncated at the experimental setup section. Per-protocol tables and the Sec. 6–7 details were not available here.