Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models

What it claims

The language sensitivity of VLAs is systematic, not task-specific, so a short rulebook can capture it and transfer to unseen tasks. Two findings matter most for anyone evaluating VLA generalization:

• Oracle phrasing nearly closes the 21-point gap between in-distribution and out-of-distribution tasks for π0 on SIMPLER Bridge. Much of what looks like a generalization gap is a phrasing gap.
• On π0.5/LIBERO, adversarial phrasing costs 50 points on in-finetune tasks and nothing measurable on out-of-finetune ones. The authors read this as the policy attending to language only where finetuning taught it to.

The fix rewrites each instruction once per episode, before the first action. It leaves the policy's weights untouched and adds no per-step inference cost.

How it works

flowchart LR
  subgraph Offline["Offline, once (8 evidence tasks)"]
    A[VLM rephrasings] --> B[Rollouts on frozen VLA]
    B --> C[Scored evidence D]
    C --> D[LLM distills 10-20 rules R]
  end
  subgraph Deploy["Deployment, once per episode"]
    I[User instruction] --> V[VLM rephraser]
    S[Scene image] --> V
    D --> V
    V --> P[Frozen VLA]
  end

1. A VLM generates many phrasings of 8 training tasks. Each phrasing is rolled out in simulation and scored by success rate.
2. An LLM distills the scored evidence into 10–20 natural-language rules. Examples from the paper: - "'coke' stays 'coke', never 'cola', 'soda', or bare 'can'" - "never swap a common noun for its sibling (keep 'plate', not 'dish')"
3. At deployment, a VLM conditioned on the rules and the first scene image rewrites the instruction. The rewrite goes to the unchanged VLA.

The rulebook is fixed before any of the 12 sealed evaluation tasks is seen.

Evidence

Main result. On the rephrase-augmented π0 (Bridge finetune), the rules improve success by 16–27% relative over no rephraser.

• Evaluated on 12 sealed SIMPLER tasks: 5 in-vocabulary, 7 out-of-vocabulary.
• Covers adversarial, VLM-generated, and human-generated phrasings. Gains are concentrated on out-of-distribution tasks.
• The effect is stable across three applier models and three independent distillations.
• For context, rephrase augmentation alone is worth about 20% relative on natural phrasings. The rules add to that.

Replication. π0.5 on LIBERO in-finetune suites goes from 93.6% to 97.8%.

Comparison with test-time verification (CoVer-style). That approach selects among phrase-action candidates at every step. In this setting it multiplies inference cost without beating the unmodified policy.

Swing patterns:

• Swapping plate for dish costs π0 19 points and π0.5 56 points.
• "Hot plate" scores 2% as a source noun and 0% as a destination.
• "Fire up the stove" succeeds 100% of the time; "turn on the stove" succeeds 6%.
• Changing "Pepsi" to "pepsi" gains 44 points, which points to token-level sensitivity.
• Specific everyday names beat generic category words: "pepsi" beats "soda" by 39 points.
• Adding a color helps out-of-vocabulary nouns on Bridge (black keyboard +42) and hurts in-vocabulary ones (yellow plate −22). On LIBERO, adding a color only hurts.

Where swings occur. Of 426 candidate pairs on π0.5, 44% of in-finetune pairs showed a significant swing of at least 18 points, versus 5% of out-of-finetune pairs.

Measurement setup:

• π0: 72 rollouts per phrase (24 layouts × 3 repetitions), with common random seeds. Success is judged at frame 60.
• π0.5: one rollout on each of 50 layouts.
• The oracle search runs 3 rounds of 16 VLM rephrasings, with candidates from gemini-pro-latest.

How to use it

The page publishes swing sets, phrases, and exact p-values at https://github.com/sttawm/vla-rephrasing-artifacts, plus a project site at https://sttawm.github.io/rephrase-before-you-act.

The fetched text gives no install commands, rulebook files, or prompts. Check the repository for those.

Adopting the method requires:

• simulation rollouts of your own frozen policy on a handful of tasks,
• one LLM distillation pass,
• a VLM call per episode at deployment.

Limits

• No multiple-comparison correction. With 426 π0.5 pairs, about 21 significant swings are expected by chance. Swings are tested with a two-proportion z-test at p<0.05.
• Manual filtering. 8 of 23 significant π0.5 sets were removed by hand as possibly scene-ambiguous.
• Some effects reverse across tasks. Adding "purple" gains 61 points on one task and loses 30 on another, so no static rule can satisfy both.
• Simulation only. All results come from SIMPLER and LIBERO, on two checkpoints inherited from prior work.
• Truncated page. The fetched text stops before the full results tables and the detailed CoVer comparison. π0.5 out-of-finetune results after rephrasing are not shown.