Skip to content

Back to the catalog

Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
arxiv.orgpaper

Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use

Critical-State RL is a method for diagnosing which model calls in a multi-turn tool-use trajectory are actually "trainable" — i.e., where reward variation reflects the action's own effect on task success rather than downstream randomness. It uses nested sampling to separate action-dependent reward signal from continuation noise, then optimizes the policy at the selected critical states via contextual-bandit training. On the Berkeley Function Calling Leaderboard (BFCL) v4, training at diagnostic-selected states (e.g., the response right after a missing tool becomes available, or right before a missing argument is supplied) improves performance by up to ~14 points, while training at naively chosen alternative states leaves performance flat or worse.

TL;DR A credit-assignment diagnostic for multi-turn tool-use RL that finds which call to train on, boosting BFCL v4 missing-function accuracy by ~14 points.
Open original ↗
Document

What it claims

Critical-State RL diagnoses which single model call in a multi-turn tool-use trajectory is worth training, then trains only there. Beyond what the summary states, the abstract makes the diagnostic an explicit two-part test on each candidate call: (1) whether the local reward actually captures that action's effect on task success, and (2) whether improvement over a reference policy is even possible at that state. A call passes only if both hold — reward variation alone is treated as insufficient because it can reflect downstream randomness.

How it works

The abstract frames the input as task-defined candidate calls with local rewards — the diagnostic does not discover candidate states on its own; the task supplies them. For each, nested sampling separates action-dependent reward variation from continuation noise, and the selected states are trained with contextual-bandit optimization. Concretely, the diagnostic's state choice is task-specific: for missing-function tasks it selects the response after the tool becomes available; for missing-argument tasks it selects the response before the missing argument is supplied.

Evidence

On BFCL v4, training the diagnostic-selected response improves the missing-function task by about 14 percentage points, while training alternative (non-selected) states leaves performance flat or worse. The abstract states this comparison as the central experiment. It also reports applying the same recipe across additional models and tasks — logged repeat-call avoidance and memory management — which the existing summary does not mention; these are named as further applications but the abstract gives no numbers for them.

Limits

This is an abstract-only page (submitted 21 Sep 2026); no code, weights, per-task numbers beyond the ~14-point figure, model names, or compute are given here. The method assumes the task provides candidate calls and local rewards, so applying it to a new setting requires defining those first — the abstract does not describe how. The gains on repeat-call avoidance and memory management are asserted without quantification, so those extensions are unverified from this page.

Organisations named here

  • Berkeley Function Calling Leaderboard

Techniques named here

  • Critical-State RL
  • Nested sampling
  • Contextual-bandit training

Related in training