Beyond Reward Hacking: Proxy Divergence Across Four Layers of a Staged Humanoid Learning Pipeline

What it claims

The paper argues that four parts of a staged RL locomotion pipeline can be "gamed" the way a reward can: the reward, the curriculum gate, the evaluation statistic and the reference motion. Each layer gets a stated divergence condition, a measured failure from one PPO lineage, and a fix. The most useful content is the set of failures that every logged metric hid, and the specific numbers behind them.

How it works

Setting. The robot is a simulated 1.91 m, 69.35 kg humanoid built from its CAD assembly.

• The policy controls 11 joints (5 per leg plus waist yaw). Arms and neck are on position drives outside the policy.
• The network is a Gaussian MLP with 3×128 ELU hidden layers and a separate critic.
• PPO settings: γ=0.99, λ=0.95, clip 0.2, adaptive learning rate at target KL 0.01, 2048 envs × 24 steps per iteration.
• Actions are offsets q* = q0 + 0.5a, run through PD control at 50 Hz over 200 Hz physics.
• Observations include a 187-ray height scan.
• All training ran on one RTX 5070 Ti laptop GPU.

Formal results:

• Reward (Prop. 1). A Gaussian tracking kernel has zero slope at rest, so any first-order incentive to drift wins. Adding an L1 term −λ|v| makes v=0 a local maximum whenever g′(0) < λ.
• Unpenalised joints. If a joint has no reward term, it settles on its limit. A quadratic regulariser pulls it into the interior.
• Dilution (Prop. 2). A gate on mean error over |J| joints accepts any single-joint miss e* < |J|(τ−ε)/ρ, where ρ is the fraction of the motion the miss lasts.
• Information. A gate that never refuses has I(G;C)=0, so it is equivalent to having no gate.
• Reachability. Run the reference open-loop as a policy. If it cannot meet τ, the gate may be unreachable.
• Phase aliasing. Synchronised resets make the fall-rate estimate converge to p(m|φ₀) instead of the phase average. The bias does not shrink with N.
• Infeasible references. If a reference cannot be executed, the optimum is a feasible trajectory near it in the reward metric. That trajectory may have a different contact sequence, and reward shaping cannot fix it.

Evidence

All results come from one lineage: four stages, 13,500 iterations.

Reward layer

• With σ=0.5, the kernel still pays 0.85 at 0.2 m/s of drift, and the policy crept forward on every zero command.
• An L1 base-velocity cost of weight 1.5 on zero commands cut drift to 0.01–0.04 m/s.
• With the waist penalty removed, the waist sat at its +45° limit for 100% of steps. The dashboards still looked better than the corrected policy: reward +27.4 vs +25.7, falls 2.8% vs 5.3%.

Curriculum layer

• The lifting hip missed its target by 59° (≈1.03 rad). The gated mean error read 0.06–0.07 rad against a 0.25 rad threshold.
• There were seven raises at 41–44-iteration intervals against a ≈41.7-iteration rate limit, with zero refusals.
• An air-time requirement of 50% exceeded the ≈30% the reference itself achieves at half amplitude.
• Restoring a checkpoint without curriculum state reset speed from 1.7 to 1.0 m/s, and it was not raised again for 2,000 iterations. Terrain level fell from 5.6 to 3.5.

Measurement layer

• A foot-height metric read 61–69% of target during training while the deterministic policy lifted nothing. The cause was ≈40° of exploration noise on the ankles (vs ≈6° at the hips) tipping the foot onto its toe.
• Synchronised push tests gave an 88% fall rate at 0.5 m/s vs 50% at 2.0 m/s, plus a spurious left–right asymmetry. Drawing each push time from U[t0, t0+1 s] made the series monotonic. Walking fall rates became 3/27/35/47% for 0.5/1.0/1.5/2.0 m/s pushes, and lateral vs sagittal came out at 30% vs 26%.
• A harness that used "any direction" instead of "every direction" reported a clean result while one direction had 38% falls.

Reference layer

• In hand-authored leg-lift references, the centre of mass lay 0.21 m outside the support foot. Seven reward, initialisation and exploration changes all failed to produce single support.
• A support-leg roll moves the centre of mass ≈0.0095 m per degree. Rolling 20° (limit 27°) left a 0.02 m residual.
• With residual feed-forward, the open-loop reference alone held the foot clear for the whole motion. A policy that had to discover the shift from reward held that joint at −1 to −3° against a −20° target.

How to use it

The page gives no code or release. Its value is as a checklist for your own pipeline:

1. Pair every rest-state kernel with an L1 cost that is active only at rest.
2. Put every actuated joint in the reward, or command a target for it (the waist here uses a ±0.7 rad target).
3. Gate on peak error over the joints the motion actually moves. Combine that with a contact-level outcome, e.g. foot force < 1 N.
4. Before training, check that τ ≤ the open-loop reference's score at each level.
5. Log gate refusals, and treat zero refusals as an unvalidated curriculum.
6. Checkpoint (weights, optimiser, curriculum level) together. Compare stages only under a fixed evaluation distribution.
7. Report outcomes from the deterministic policy (μθ), not from noisy training rollouts.
8. Randomise each robot's disturbance time over a window longer than one gait period. Use min over directions for universal claims.
9. Check that the centre of mass falls inside the support polygon at every keyframe. Feed the reference forward as q* = q0 + 0.5a + r(t). The action space is unchanged, so existing checkpoints still load.
10. To add observations without retraining, append the new inputs and zero-pad the first-layer weights of actor and critic.

Limits

• Evidence is a single policy lineage on one simulated robot. These are illustrations, not controlled comparisons across seeds or robots, and there is no hardware or sim-to-real result.
• The fetched text stops partway through Section VIII. The full policy-growth method, including the "split authority" part, and any results for it are not available here.
• Even after the fix, fall rates are high: 10% while standing at the training push magnitude of 0.5 m/s, and 62% at 2.0 m/s.