Vision-Language Models Are Joint Denoisers in Robotic Action Generation

What it claims

JADE puts the vision-language model inside the denoising loop of a π0.5-based flow-matching VLA. At every step the VLM reads compact latents of the current noisy action chunk and steers the action expert. The authors (Peking University, HKUST Guangzhou, Microsoft Research Asia) say this extra per-step cost is largely paid back because joint denoising needs far fewer sampling steps. Code is at https://github.com/gxccc123/JADE.

How it works

• Action interface. A frozen OAT encoder (Liu et al., 2026) turns the noisy action into M continuous register latents. JADE uses the pre-quantization values, not OAT's FSQ codes. OAT is pretrained per benchmark by action reconstruction. The setup uses eight four-dimensional registers.
• VLM pass. A trainable adapter maps the latents into the VLM. The registers attend to the observation prefix (images, instruction, state) and to each other. The prefix does not attend back to the registers, so its keys and values are computed once per policy query and reused across all denoising steps. A bottleneck residual adapter, shared across layers, refines the register states.
• Fusion. Each action-expert layer gets a new cross-attention block between native attention and the FFN. It reads that layer's VLM register features. A tanh gate controls the residual and combines a per-layer bias, a shared gating network and a noise-level embedding. The gate is zero-initialized, so training starts from unmodified π0.5 behaviour.
• Loss. The training loss is native flow matching plus λ_align times a SmoothL1 loss. In that term a linear head predicts the clean OAT latents from the final VLM register features. Both terms train the VLM. OAT stays frozen.
• Adaptive sampler. The sampler keeps EMAs of the time-weighted correction q_i = −τ_i·v_i. The step size is λ_step/(c_i+ε), where c_i is the EMA standard deviation. It is clipped to [h_min, h_max] and capped by the remaining τ. Stable corrections give large steps. The schedule needs no extra model evaluations and leaves the weights unchanged. By default JADE uses uniform Euler steps, like π0.5.

flowchart LR
  P[Images + instruction + state] --> C[Cached prefix KV]
  A[Noisy action] --> O[Frozen OAT latents]
  O --> V[VLM registers]
  C --> V
  V -->|gated cross-attn| E[Action expert]
  A --> E
  E --> S[Euler step, adaptive h]
  S -->|next step| A

Evidence

The fetched text covers only the setup, not the result tables. RoboTwin 2.0 trains one multi-task policy on clean demonstrations of 50 bimanual tasks. It is tested on Clean scenes and on Random scenes not seen in training. RoboDojo trains on 3,500 demonstrations and tests 42 tasks in five capability categories, reporting success rate and the official score. The base model uses three RGB views and a 50-step action chunk.

The authors say other action representations besides OAT also beat π0.5 (Section 4.3). They pick OAT because it scored best and its few registers keep the per-step VLM cost low. Ablations and the latency trade-off (Section 4.4) are referenced but not in the fetched text.

Limits

• The page text ends partway through the implementation details. No RoboTwin numbers, latency or step-count figures, ablation values or real-robot task descriptions appear.
• The full VLM runs at every velocity evaluation. Prefix caching helps, but total cost still grows with step count.
• OAT must be pretrained separately for each benchmark's action space.
• The fetched text states no licence and no released weights.
• The authors state that c_i guides sampling only and is not a task-difficulty or success estimate.