Skip to content

Back to the catalog

Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
arxiv.orgpaper

Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient

This paper introduces STL-SVPG (Signal Temporal Logic-guided Stein Variational Policy Gradient), a population-based reinforcement learning method that uses smooth STL robustness as a trajectory-level training objective instead of conventional dense reward shaping. By differentiating the STL objective through system dynamics, the method assigns credit to policy actions based on their effect on the full task specification — capturing event-triggered responses, strict ordering, deadlines, and physical contact — rather than local progress alone. Evaluated on six quadcopter and manipulator tasks, STL-SVPG achieves the highest success rate on five of six benchmarks and transfers temporal/contact behavior from simulation to real robots.

TL;DR A new STL-guided Stein Variational Policy Gradient method beats reward-shaping baselines on sparse-success robot tasks and transfers sim-to-real.
Open original ↗
Document

What it claims

STL-SVPG trains a population of closed-loop feedback policies by backpropagating a smoothed Signal Temporal Logic robustness score through a differentiable simulator. Because the score is computed from the same specification that defines success, a rollout that fails still produces a gradient. The paper contrasts this with STL-SVPIO, its open-loop predecessor. STL-SVPIO only optimizes control sequences from fixed initial conditions and is too slow to replan online. STL-SVPG amortizes that computation into a policy. The comparison baselines are RL methods with dense rewards generated by an LLM-coding-agent reward-design approach.

How it works

  • Objective. The objective is expected robustness J(θ) over randomized initial conditions and task inputs. The target distribution over policy parameters is a Gibbs density p*(θ) ∝ exp(J(θ)/α). Minimizing KL to this target is equivalent to maximizing expected robustness plus entropy. That entropy is over policy parameters, not over actions.
  • Smoothing. Max and min are replaced by LogSumExp with sharpness β, which is separate from the temperature α. A smaller β spreads gradient across more timesteps. The authors cite the problem this addresses: exact robustness gradients concentrate on a few critical steps and vanish over long horizons. Smoothed robustness is used only in training. Evaluation uses exact robustness.
  • Simulator. Gradients flow through MuJoCo MJX in JAX with soft contact models.
  • Per-particle gradient, as stated in Algorithm 1: 1. Roll out E episodes per particle. 2. Store ∂ρ̃/∂u_t, the gradient of smoothed robustness with respect to each action. 3. Multiply by the policy action Jacobian ∂u/∂θ and sum over time. 4. Accumulate over randomized transition minibatches, so the simulator is not re-differentiated each time.

Observations and policy noise are held fixed during this step. The result is therefore a local approximation, not the full closed-loop derivative.

  • Kernel. Particle similarity is measured in action space: an RBF kernel on the concatenated actions of each policy over a shared random observation batch. Parameter distance is not used. The Stein update gives kernel-weighted sharing of each particle's gradient plus a repulsive term for diversity.

Evidence

  • Platforms. Most tasks run on a Crazyflie quadcopter, simulated with a cascaded body-rate PID, first-order motor dynamics and rotor drag. A Franka Emika Panda is used to show the method carries over to a second embodiment.
  • Task types. The tasks involve event-triggered responses, strict ordering, deadlines and contact.
  • Results. STL-SVPG has the highest mean success rate on 5 of 6 tasks.
  • Real hardware. Sim-only policies transfer zero-shot on two tasks. One is a drone ping-pong task: hit an incoming ball into a target.
  • Ablations. The paper says it analyzes robustness-gradient guidance and population interaction separately, and claims an advantage over plain gradient descent without SVGD.

How to use it

To apply this you need:

  • a differentiable simulator of your robot (they used MJX/JAX with soft contact);
  • a hand-written STL formula for task completion;
  • these hyperparameters: N particles, E rollouts per particle, horizon H, α, β, kernel bandwidth h, and step size.

The fetched text gives no code link, hyperparameter values or compute cost.

Limits

  • The fetched text stops inside Algorithm 1. Per-task success numbers, baseline margins, the one task where STL-SVPG loses, ablation figures and the authors' own limitations section were not available. Treat "5 of 6" as an abstract-level claim.
  • The policy gradient deliberately ignores how actions change future observations. How much this matters on long, contact-heavy horizons is not shown in the available text.
  • The method depends on gradients through soft-contact simulation. Tasks that need hard contact or a non-differentiable simulator are outside what is described.
  • You must still write the STL specification. That replaces reward engineering; it does not remove engineering effort.

Organisations named here

  • Signal Temporal Logic
  • Stein Variational Policy Gradient
  • STL-SVPG

Techniques named here

  • STL-Guided Stein Variational Policy Gradient

Related in research