PEARS: Physical-Prior-Guided Efficient Adaptation via Failure Reasoning and Diffusion Steering for Tactile Manipulation

What it claims

PEARS adapts a frozen pretrained manipulation policy (GR00T N1.7 fine-tuned as a VLA, or a VTLA with fingertip force-torque input) to out-of-distribution conditions online. It splits the problem in two. Latent-noise RL fixes where and when the robot makes contact. A VLM diagnoses failures between episodes and sets how hard the robot presses. The authors say this split, rather than stochastic RL exploration alone, is what cuts interaction cost.

How it works

The page describes several details the summary does not:

• Base policy. GR00T N1.7 is fine-tuned on task demonstrations with two 224×224 images, one wrist and one third-person.
• The proprioceptive vector is 10-D: position, 6D rotation, and gripper width.
• The VTLA variant appends left and right fingertip 6-D wrenches, giving 22-D.
• Actions are chunks with horizon H=32. The robot executes the first K=10 steps, then replans.
• Latent actor. Visual tokens are reused from the frozen VLM backbone and pooled to 2048-D.
• A Transformer fuses these with proprioception and tactile input and outputs a 10-D noise vector.
• GR00T expects noise of shape H×132. The 10 values fill the first 10 channels at every chunk position. The other 122 channels come from one Gaussian sample that is held fixed.
• Actor and critic therefore both work in a 10-D action space.
• RL algorithm. The setup follows DSRL-SAC, with one transition per executed chunk.
• It uses a critic ensemble with a min-over-ensemble TD target, without entropy backup.
• The entropy coefficient is tuned automatically.
• The critic sees the current force goal as extra context, z=(o, g). The actor sees only o. The authors frame this as a POMDP.
• Force control.
• Axis choice (before training). A VLM proposes the force-control axis, and a human corrects it if wrong. At most one object–environment axis is regulated, plus grasp force if needed, so m≤2.
• Bound updates (after each episode). PFR sends the VLM the final contact image, the force trajectory and the current bounds. The VLM labels the force as under, over or adequate, separates force failures from positioning or tracking errors, and shifts the interval.
• Example output from the figure: "failure_cause": "Under-force", "force_range": [8.95, 10.95], "confidence": 0.87.
• Execution. Bounds stay fixed within an episode. A tactile-gated hybrid force–position controller enforces them only during the contact phase. The figure shows control releasing when the force-corrected action deviates from the VLA action by more than Δa_max.

flowchart TD
  O[Obs: images, proprio, wrenches] --> A[Latent actor → 10-D noise]
  A --> P[Frozen GR00T policy]
  P --> C{In contact?}
  C -- no --> X[Execute position action]
  C -- yes --> F[Hybrid force–position ctrl]
  G[Force bounds] --> F
  X --> E[Episode ends]
  F --> E
  E --> V[VLM failure diagnosis]
  V --> G
  E --> R[SAC update on chunks]
  R --> A

Evidence

The visible text gives only the abstract-level numbers: +12.4–37.4 points over the strongest per-task baseline, up to 53.2% fewer episodes than the fastest baseline, and 95% and 90% success on the two real tasks.

• Simulated tasks. The figure names thin-sheet transfer, fragile fruit picking and bottle-cap twisting.
• Baselines. The related-work section suggests the comparison set is DSRL, RL Token, TORL-VLA and similar methods, but the visible text does not say which were run.
• Missing. The fetched text cuts off in Section IV. Per-task tables, baseline identities, episode counts and ablations are not visible.

How to use it

To reproduce this setup you would need:

• Hardware: a gripper with per-fingertip 6-D force-torque sensing.
• Base policy: a flow-matching or diffusion base policy whose initial noise you control. The paper uses GR00T N1.7.
• VLM: access to a VLM for between-episode queries. The visible text does not name which VLM.
• Human step: a person to verify the force axis for each task.

The project site is https://song-kun.github.io/pears. The visible text mentions no code, weights or licence.

Limits

• Manual setup. Each task needs a human to check or correct the VLM's force-axis estimate.
• Narrow force regulation. Only one wrench axis plus grasp force is regulated, so tasks needing multi-axis force control are outside this design.
• Restricted steering. Steering covers only 10 of GR00T's 132 noise channels, repeated identically across the chunk. That limits how much any single chunk can be steered.
• Slow force adaptation. Bounds update only between episodes, so force mistakes within an episode are not corrected.
• Small real-world validation. Only two real tasks were tested. The page is truncated, so check the full paper for ablations of PFR versus DSRL alone before relying on the efficiency claim.