Shockingly Simple Self-retrospection Improves Agentic Models Without RL

What it claims

ROFT (Retrospection-Only Fine-Tuning) improves Qwen3.5-4B on agentic coding by fine-tuning only on the model's own post-attempt explanations. It reaches higher held-out SWE-bench solve rates than GRPO at roughly half the compute per update. The base model scores 44.2% on SWE-bench Verified and 23.9% on SWE-bench Pro. ROFT at 20 updates reaches 49.2% on Verified. GRPO's best checkpoint reaches 48.0% at 40 updates. The paper is a September 2026 arXiv preprint from Microsoft Research with RPI, UC San Diego, KAIST and Mila authors.

How it works

Each training step repeats one loop:

flowchart LR
  A[Attempt task] --> B[Trajectory + test feedback]
  B --> C[Same model writes K retrospections]
  C --> D[NTP loss on retrospection tokens only]
  D -->|updated weights| A
  B -.masked, context only.-> D

• Retrospection prompt. The default prompt asks the model for four things: a consequential assumption or decision, the evidence for or against it, a correction if needed, and concrete triggers for applying the lesson. Both passing and failing attempts are used.
• Loss. Cross-entropy is averaged over the retrospection tokens. The task, actions, observations and feedback are masked as targets.
• What is left out. There is no action-target loss, no reward weighting and no quality filter on the retrospections.
• Evaluation. Later attempts, including evaluation, see no stored retrospection and take no reflection step. Any gain has to come through the weights.
• Combination with GRPO. ROFT can be added to GRPO by appending a retrospection step to each rollout.

Evidence

Setup. Training uses SWE-rebench-767: 767 SWE-rebench problems on which Qwen3.5-4B had mixed outcomes across three attempts. The authors say this selection favors GRPO. Both methods ran on four training GPUs and four inference GPUs.

Held-out SWE-bench Verified:

GRPO's Verified score falls from update 40 to update 90 while its training reward keeps rising. The authors read this as overfitting.

Cost. For 40 updates, ROFT took 4.32 h and 6,035 solution attempts. GRPO took 8.30 h and 11,576 attempts.

SWE-bench Pro. GRPO reaches 25.3%. ROFT is reported as 26.8% in the abstract and Figure 1, but as 29.0% at 20 updates in Section 3.1.

Beyond the frontier. The authors trained separate runs on single tasks where 0 of 64 base attempts succeeded. GRPO gets no gradient in this case. After 40 ROFT updates, the online solve rate was 1.75% on a SymPy task and 3.29% on a SymbiFlow task. Single-task training did not significantly degrade test-set performance.

Behavior. On labeled base-model trajectories, likelihood increases were more selective for correct turns than for incorrect ones. The authors take this as indirect credit assignment. Asking retrospections to emphasize more direct solutions produced shorter later attempts, with no length penalty.

Difficulty profile. Over 16 attempts on SWE-bench Verified, the base model solved 10.4% of problems every time, 61.8% sometimes and 27.8% never.

Limits

• Scope. Results cover one model (Qwen3.5-4B) and one domain (software engineering). The abstract describes the numbers as "in the evaluated runs", and the text shows no variance across seeds.
• Pro number. The discrepancy between 26.8% and 29.0% needs checking against the full paper.
• "No verifier". This means no reward in the objective. Test feedback is still part of the context the retrospection is written from.
• Saturation. ROFT's training curve levels off around 20 updates. The authors hypothesize that retrospections become repetitive on revisited tasks, consistent with falling reflection entropy and SFT loss.
• Small frontier gains. Beyond-frontier success rates are low in absolute terms and come from single-task runs.
• Missing details. The fetched text is truncated partway through the results. It does not mention released code, weights or a licence. Sampling settings, the batch construction and prior-method baselines are in appendices that were not fetched.