Proxy Policy Steering

How it works

The summary names two proxies but not what each is trained to do. The page: the reference proxy models the frozen base's behavior on the target-task observations; the task proxy is initialized from the reference and then trained under task supervision, so it captures only how behavior shifts once task demonstrations are added. Their difference is a calibrated velocity-space residual applied to the frozen base sampler at every denoising step. The authors claim to identify the conditions under which this residual isolates the change induced by task supervision (rather than incidental differences), and say they validate those conditions empirically — a theoretical contribution the summary drops.

flowchart LR
  O[target-task observations] --> R[reference proxy: models frozen base behavior]
  O --> T[task proxy: init from reference, trained on demos]
  R --> D{velocity-space difference}
  T --> D
  D -->|residual, every denoising step| B[frozen base sampler]
  B --> A[steered action]

What it adds beyond the summary

• No base-parameter access required. Adaptation needs only forward velocity predictions from the base — so PPS is applicable even when you cannot touch the base weights (e.g. a served/closed policy), not merely "chooses not to." This changes who can use it.
• Zero-to-one gains. The 53% average lift includes tasks the base never solves going to non-zero success — the summary's aggregate number hides this qualitative case.
• Preserved emergent behavior. Because the base is untouched, behaviors the demonstrations don't exercise — the page cites recovery from failure specifically — remain available at inference. This is the concrete argument for steering over fine-tuning.
• One more baseline. Beyond LoRA, from-scratch specialists, and residual policies (in the summary), the page states PPS also beats prior inference-time steering methods — the most relevant comparison class, and the one the summary omits.

Limits

This is an abstract-only page; no per-task numbers, no ablation figures, no baseline breakdowns, and no code or weights link are given here — the 53% is an average across the 12 tasks with no spread reported. The "conditions under which the residual isolates the task change" are asserted as identified and validated but not stated on this page. Latest version is v3 (13 Sep 2026); the summary and this analysis reflect v1's abstract. Verify the actual result table and whether code is released before adopting.