Learning to Steer LLM Agents and Multi-Agent Systems

What it claims

A separately trained steering LLM watches a frozen executor (a single agent or a multi-agent system) and intervenes mid-run when the executor drifts from the task goal. Trained steering helps much more than prompting a model to steer:

• Single agent, seven QA benchmarks: zero-shot steering adds 1.84 points. The RL-trained steerer adds 6.66 points while averaging only 0.93 interventions per task.
• Highest intervention frequency: raising the trigger rate, with no retraining, lifts the gain to 17.11 points.
• Five-role multi-agent system: zero-shot supervision makes results worse. The RL-trained steerer adds 5.34 points.

The page reports these as "points" and does not name the metric.

How it works

• What it sees and does. The steerer gets an observation of the goal plus the execution context. Its actions are pass or an intervention the host system supports; the figure shows note and retry. What it observes sets how fine-grained the supervision is:
• Every step of a single agent gives fine-grained supervision.
• Only inter-agent handoffs gives coarse-grained supervision of a multi-agent system.
• Fast path. One forward pass reads the log-odds of yes vs no and turns them into a "steering priority" between 0 and 1. The priority estimates how much intervening now would change the final outcome.
• Trigger. The slow path fires with a probability set by a soft threshold τ on that priority. Changing τ is how intervention frequency is adjusted without retraining.
• Slow path. The same LLM backbone reasons step by step and then emits an intervention action.
• Training. The setup is treated as a partially observable decision process, with the frozen executor acting as the environment and a reward given only at task end.
• States are sampled from earlier rollouts, and several candidate steering actions are sampled at each one.
• The fast path is trained with a pairwise ranking loss over states. The slow path is trained with GRPO.
• Against the alternatives the paper names:
• Planning and post-hoc verification can catch drift only after the effort is spent.
• SupervisorAgent prompts an LLM and targets easy faults such as errors and repeated actions.
• Multi-agent RL fine-tunes the executors themselves, which is costly and can overfit.

flowchart LR
  O[Observation update] --> F{Fast path: priority vs τ}
  F -- below --> P[pass]
  F -- above --> S[Slow path: reason + act]
  S --> A[note / retry / ...]
  P --> E[Frozen executor resumes]
  A --> E
  E --> O

Evidence

• Drift patterns the paper targets:
• Redundant searching: a single agent keeps searching the same film after it already has the actress it needed.
• Over-rejection: in the multi-agent system, a verifier keeps rejecting correct answers, which forces extra revision rounds.
• Multi-agent setup: five roles (planner, two searchers, verifier, summarizer), taken from Xu et al. 2026.
• Benchmarks: the tasks are information-seeking search; GAIA and 2WikiMultiHopQA are cited. The seven QA benchmarks are not listed in the fetched text.
• Generalisation: the paper says the steerer works on out-of-domain tasks and transfers to executors it was not trained with. No numbers for either appear in the fetched portion.

Limits

• The fetched text stops partway through the training section. The full RL details, ablations, executor models, backbone size and compute are missing.
• No code, weights or licence are mentioned.
• Only search and QA tasks are evaluated. Software-engineering agents are named as motivation but not tested.
• The 17.11-point gain at maximum frequency comes with no reported cost in extra interventions or tokens.
• Zero-shot steering hurting the multi-agent system means you should not expect to get this benefit by prompting an off-the-shelf model as a supervisor.
• This is a v1 preprint dated 15 Oct 2026.