SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories

What it claims

A 30B-A3B MoE verifier (fine-tuned from Qwen3-30B-A3B-Thinking-2507) reads one coding-agent trajectory and outputs a YES/NO "issue resolved" judgment. Before the judgment it writes a structured rationale. The 57% → 66% → 78% headline is on a test set where about 66% of trajectories are successful (28,741 of 43,445, from the paper's own counts). Always answering YES would score about 66%, so SFT alone only matches that baseline. The real gain is from RL, and it is uneven across task sources.

How it works

flowchart LR
  D[CoderForge-Preview<br/>filtered] --> T[Test: 20% of issues]
  D --> S[SFT: 1 pass + 1 fail per issue]
  D --> R[RL: class-balanced rest]
  S --> G[GPT-5.5 pairwise<br/>differential CoT]
  G --> F[SFT checkpoint]
  F --> P[GRPO RL]
  R --> P
  U[Unit-test outcome] --> P

• Data. The source is CoderForge-Preview: 258K trajectories (155K pass, 103K fail) over 51K issues and 1,655 repos. All were generated by OpenHands with Qwen3-Coder-480B on R2E-Gym, SWE-Rebench and SWE-Smith. Trajectories over 64K tokens or with no submitted patch were dropped, which removed 16% of the data. The test split is by issue, so no test issue appears in training.
• CoT annotation. GPT-5.5 compares a successful and a failed trajectory for the same issue. Its output is then post-processed into pointwise rationales with three parts:
• a procedure summary;
• mistake tags from a fixed taxonomy (13 named categories, e.g. incomplete code, edge cases, API constraints, harness integrity, plus "others");
• a causal summary.
• SFT. Uses 10,390 annotated trajectories, which is 8.7% of the training data. It is standard cross-entropy on the CoT plus the judgment.
• RL. Runs on the remaining 91.3%, which has no CoT. It uses GRPO with group size 8 and a KL penalty to the SFT model. The reward is +1 if the judgment matches the unit-test result and −1 otherwise. There is no process reward.

Evidence

All numbers below are from the 43,445-trajectory held-out test set, with one sample per item.

• Base model. It almost never says NO: negative recall is 0.00 (51 of 14,704 failed trajectories caught). In addition, 9.6% of its responses break the output format.
• SFT. It fixes the format problem completely. Negative recall rises to 0.62, but positive recall drops from 0.85 to 0.67.
• RL. It brings positive recall back to 0.85 while keeping negative recall at 0.64. Overall F1 is 0.84 for the positive class and 0.66 for the negative class.
• On R2E-Gym, both trained models score below the always-YES rate.

How to use it

• Released artifacts. The SFT and RL checkpoints, the data splits and the training code are on GitHub and Hugging Face. The page does not give repository names or a licence.
• Prompt and output format. The prompt is the task plus the full trajectory. The output follows Qwen's thinking template: <think>…</think> followed by a <judgment>YES/NO</judgment> tag.
• Context limits. Inputs are capped at 64K tokens. Training used a max prompt of 68K and a max response of 60K.
• Compute (all on Huawei Ascend; no GPU recipe is given).
• SFT: MindSpeed-LLM on 128× 910B NPUs, 3 epochs (975 steps), about 15 hours.
• RL: VeRL on 128× 910C NPUs, 1 epoch (3,396 steps), about 533 hours.
• Inference: vLLM Ascend on 16× 910C.

Limits

• Preliminary evaluation. Only YES/NO accuracy was measured. The quality of the natural-language explanations was not evaluated. There is no comparison against other verifiers, only against the untuned base model.
• Weak at catching failures. Negative recall is 64%, and the authors say failed trajectories remain hard to identify. Before using it as a gate, check what share of failures it would let through.
• R2E-Gym is weak. The authors attribute this to its smaller data volume.
• Single generator. Every trajectory came from one agent harness (OpenHands) and one model (Qwen3-Coder-480B). Transfer to other agents is untested and listed as future work.
• Length mismatch. The paper motivates the work with trajectories of hundreds of turns and millions of tokens, but anything over 64K tokens was excluded. The verifier is untested on exactly the long-horizon cases it was motivated by.