Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models

What it claims

Progress Probes (P²) is a runtime failure monitor for vision-language-action policies. It reads task progress linearly out of the policy's own activations, with no external model. On π0.5 it reports AUROC of 0.788 across 6 VLABench simulation tasks and 0.78 across 6 real-world manipulation tasks. The authors (Carnegie Mellon and Cornell, arXiv October 2026) say this matches or exceeds SAFE, which trains on failure data, and Robometer, a VLM-based progress estimator. They report the probe runs between 5× and 20,000× faster than an external foundation-model monitor.

How it works

Every timestep of a successful demonstration gets a progress label of normalized time remaining, ζ_t = 1 − t/T. The label runs from 1 at the start to 0 at the end, so no human annotation is needed. Activations come from one transformer layer of the PaliGemma backbone, mean-pooled across tokens into a single d-dimensional vector. A linear regression ϕ(z) = wᵀz + b is fit on these vectors to predict ζ.

At runtime, the expected progress is 1 − t/μ_T, where μ_T is the mean length of the successful demonstrations. The monitor score is the predicted progress minus this expected progress. A positive score means the policy is lagging, and a failure is flagged once the score exceeds a threshold δ. Running ahead of schedule is never penalized. The threshold is calibrated on a 20% validation split of the success data, following FIPER. The probe reuses activations already computed in the policy's forward pass, so the extra compute is one dot product.

Evidence

In simulation, π0.5 is fine-tuned on 5,000 VLABench trajectories (500 per task, 10 tasks). Monitoring is evaluated on the 6 tasks where the policy has non-negligible success. The probe trains on 300 successful trajectories (50 per task). Evaluation uses 600 rollouts (100 per task), split evenly between successes and failures. Failure onset is labeled by hand for frame-level scoring.

The baselines fall into three groups:

• Input-level: FIPER RND-OE and FAIL-Detect logPO on DINO features.
• Feature-level: FAIL-Detect logpZ0, SAFE-MLP and SAFE-LSTM.
• Output-level: FIPER ACE and STAC.

Robometer is tested zero-shot and fine-tuned. SAFE and fine-tuned Robometer get 300 additional failed trajectories that P² does not use. Latency was measured on an NVIDIA RTX A6000.

Limits

The fetched text stops partway through the experiments section. Per-baseline AUROC, detection-time results, the generalization to unseen tasks, the VLM-feature probe comparison and real-world setup details are not in it. Only the headline 0.788 and 0.78 figures are available.

Only π0.5, with its PaliGemma backbone, is studied. The text does not say which layer is probed. The nominal schedule assumes successful runs take roughly the mean demonstration length. Tasks with highly variable successful durations could therefore trip the monitor, though the authors do not test this. Failures that keep pace with the schedule while doing the wrong thing would not raise the score.

The text mentions no code release, weights or licence.