AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

What it claims

Post-training a 9B planner with supervised fine-tuning (SFT) and multi-turn RL closes most of the gap to frontier coding agents on robot manipulation. The two stages contribute very unequally. On LIBERO-PRO with the RPent harness (VLA skill calls), SFT alone lifts success from 18.1% to 65.1%, and RL adds the rest, reaching 71.0%, all without external memory. The same checkpoint scores 72.2% on LIBERO-Plus (10,030 configurations) with no extra training, against 19.2% for the base model. On the control-code harness CaP-X, success rises from 2.7% to 28.5%. Figure 1 labels the base planner Qwen3.5 9B. It also shows Claude Code (Opus 4.7) with memory at 82.4%, so the 9B model is close to Codex (72.1%) but not to the best frontier result.

How it works

Only the planner is trained. Perception modules, motion planners and VLA executors stay frozen.

• The model-call boundary. The shared interface sits here, not at low-level actions. A proxy records each request's context, generated token IDs and rollout log-probs. This matters because harnesses prune images or drop earlier reasoning between turns. The trainer re-renders each prompt and gives zero weight to any sample whose prompt tokens do not match the recording.
• Per-harness adapters. Each adapter is plain conversion code that reports five fields per episode: task, initial-state ID, attempt index, reward and a validity flag. Task failures count as reward 0. Infrastructure failures are excluded from training.
• SFT data. Training examples come from successful teacher trajectories.
• For RPent, the teacher had a memory module and the student does not. Memory calls are stripped from the student's context. A response is flagged as memory-dependent if it uses memory keywords, repeats 8 consecutive words of memory text, or contains a number found only in memory. Flagged reasoning is rewritten and kept only if a judge labels it OK. Tool-call arguments are never changed.
• For CaP-X, the teacher's outputs are re-rendered with the student's chat template and used directly.
• RL. The reward is terminal task success only: no intermediate rewards and no length penalty. Advantages are RLOO leave-one-out across attempts with the same task and initial state. There is no critic and no standard-deviation normalisation. The loss is dual-clipped PPO with a correction for rollout/trainer probability mismatch. At most 16 responses per episode are trained. Weights are fixed during each collection round.

flowchart LR
  P[Planner] --> PX[Recording proxy]
  PX --> H1[RPent: VLA skills]
  PX --> H2[CaP-X: control code]
  H1 --> A[Adapter: reward + validity]
  H2 --> A
  A -->|valid| G[RLOO groups]
  A -->|infra failure| X[Excluded]
  G --> U[Dual-clip PPO update]
  U --> P

How to use it

The authors say they will release the framework and planner checkpoints. No repository contents, commands or licence appear on the page.

To adopt the approach for a new harness, you need two things:

• an adapter that emits the five episode fields above;
• a serving setup the proxy can intercept to capture token IDs and log-probs.

The figure suggests a 27B teacher from the same Qwen family generated the SFT trajectories, but the label text is garbled.

Limits

• The fetched text stops at the start of the experiments section. Training compute, dataset sizes, the memory-handling ablation (Section 4.5) and per-task results are not visible.
• Most of the gain comes from SFT on teacher data (+47.0 points). RL adds 5.9 points.
• The CaP-X result is far lower in absolute terms (28.5%).
• Everything reported is from simulation benchmarks (LIBERO-PRO, LIBERO-Plus). No real-robot results appear in the fetched text.