InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

What it claims

A fixed, pretrained humanoid controller can be adapted to unseen loco-manipulation tasks by searching only over the task specification. The tasks include tipping or rotating a box in place, kicking a box with the feet only, and carry-place-kick compositions. The controller's weights never change. Each candidate reward costs a batch of parallel rollouts instead of an RL training run, which is the stated difference from Eureka-style LLM reward design. The method is built on BFM-Zero's forward-backward (FB) model, extended to observe objects.

How it works

Object-aware FB model. BFM-Zero observes only the body, so rewards that differ only in where an object should go collapse into one behaviour. All three networks (actor, forward map, backward map) keep their pretrained body branch frozen and add a trainable object residual. For the actor, the mean action is tanh(μ_prior(o_body, z) + Δμ(o, z)). The object features are:

• position, orientation, linear and angular velocity;
• distance-decayed vectors from body links to the nearest object surface.

The residuals are trained on large-scale human-object interaction data.

Reward → latent prompt. A bank of body-object states is sampled once from the motor model's multi-task replay and then frozen. B(s) and the reward features are cached for every bank state. At every control step, the active stage's reward is scored over the bank. The prompt z is the projection of a reward-weighted sum of B(s), with weights ∝ r·exp(βr). This tilts z toward high-reward states while the rest of the bank keeps broad coverage. Because rewards read live context such as contact and object pose, z is recomputed each step.

Reward program. A program is an ordered list of stages, each with:

• reward code;
• a completion condition (object pose, contact status, stage time);
• a shared constant vector θ holding weights, tolerances, kernel widths and thresholds.

Search.

flowchart TD
  T[Task + scene] --> P[LLM agent edits structure]
  H[Skill library] --> P
  P --> C[CMA-ES tunes θ]
  C --> R[Parallel rollouts, 192 envs]
  R --> V{Fixed verifier}
  V -- criterion-level feedback --> P
  V -- beats current best --> B[Replace best]
  B -- final, verified --> H

• Fixed verifier. It is instantiated once from the language request by an LLM-assisted parser, then held fixed so the agent cannot ease the task by rewriting it. Criteria include upright, hands-off and object-moved checks.
• Feedback. Feedback to the agent is per-criterion pass counts with diagnostic framing. One example from the kick task: "hand_frac 170/192", flagged as "a TAIL/VARIANCE failure, not a low mean", with "only 93/192 envs reach stage 1".
• Two loops. The outer loop revises program structure from this evidence. The inner CMA-ES loop tunes θ for each proposal.
• Skill library. Each task's final program and verifier results are stored for retrieval on later tasks.

Evidence

The fetched text cuts off before the experiments section, so no success rates, baselines or ablations are available here.

The only result statements come from the abstract and Figure 1:

• On an in-place box rotation absent from training data, a human-designed reward fails and the evolved program succeeds.
• Evolved skills run on a physical Unitree G1 from onboard egocentric perception.

Limits

• No quantitative results in the fetched text. The page is truncated mid-method. Read the full paper before relying on any performance claim.
• Privileged state. Rewards and the verifier read privileged body-object state in simulation. The excerpt does not describe how this maps to onboard perception on the real robot.
• Capability ceiling. The approach only repurposes competence the controller already has. Tasks outside the learned motor repertoire are not addressed.
• Budget and cost not given. The excerpt gives no round budget K, no LLM used, and no wall-clock or compute cost per task.
• No release details. No code, weights or licence are mentioned. The only link is the project page, sirui-xu.github.io/InterEvolve.