Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

What it claims

A base checkpoint's agentic-coding potential can be ranked before post-training. The probe is built at the decisive step of a successful frontier-model trajectory: the first code-changing step whose cumulative patch makes the task's tests pass. The cohort is ten public base/post-trained pairs, ranked against post-trained SWE-bench Verified pass@1. Spearman ρ for each screen:

• Decisive-Action BPB: 0.964, on trajectories from DeepSWE, a separate benchmark.
• Patch MCQ: 0.903, on SWE-bench Verified trajectories.
• Prefix-conditioned pass@16: 0.988, on SWE-bench Verified trajectories.

Existing screens are inconsistent on the same cohort. RepoBench XFirst reaches ρ = 0.830, but HumanEval comes in at −0.394. End-to-end pass@K on base models stays near zero because untuned base models cannot operate the tool harness. The authors are from NVIDIA, with UMN and UC Berkeley.

How it works

flowchart TD
  A[Successful agent trajectory] --> B[Replay next code-changing step]
  B --> C{Verifier passes cumulative patch?}
  C -- no --> B
  C -- yes --> D[Decisive step T*, golden action y*]
  E[Candidate actions sampled from post-trained LLMs] --> F{Verifier on prefix + candidate}
  F -- fail --> G[Distractors]
  D --> H[Decisive-Action BPB]
  D --> I[Patch MCQ]
  G --> I
  D --> J[Sample K continuations from prefix, verifier judges]

• Trajectory collection. Trajectories come from a frontier post-trained model in a harness such as mini-swe-agent. The evaluation context contains only the steps strictly before T*.
• Decisive-Action BPB. This is the negative log-likelihood in bits over the tokens fully inside y\, divided by y\'s byte length.
• All scoring uses one fixed universal chat template.
• Template formatting tokens are masked out, because base models may be unfamiliar with the template (Appendix H.1).
• Lower BPB is better.
• Patch MCQ.
• Distractors: candidate actions sampled from post-trained LLMs at the same prefix that fail the verifier.
• Scoring: answer-letter likelihood over the full option set.
• Why it exists: BPB cannot separate two near-identical actions where a one-character difference decides pass or fail.
• Prefix-conditioned pass@K. The base model generates K continuations from T\*. Any continuation the tests accept counts as a success, not only the recorded action.

Evidence

Decisive step vs. all steps. Restricting BPB to the decisive step instead of averaging over every step raises ρ from 0.915 to 0.964. It also corrects two pairwise mis-rankings made by all-step BPB:

• DeepSeek V4 Flash vs. Nemotron Ultra 550B
• Kimi K2 vs. Qwen 3.5 35B

Prior MCQ benchmark audit. The authors checked 1,682 APTBench SWE MCQs. They found items whose designated correct answer is wrong or not uniquely correct (Appendix A). Appendix B reproduces APTBench's correlations on this cohort. Their argument is that aggregate correlation can hold even when the MCQ items are defective.

How to use it

You need three things:

• successful trajectories for a benchmark
• the benchmark's executable verifier
• a restorable environment state at T\*

The recipe is generic and can convert any agentic coding benchmark into a base-model evaluation. The fetched text gives no code release, install commands, or dataset artifact.

Limits

• Small cohort. All validation rests on ten model pairs.
• Single target benchmark. The downstream target is only SWE-bench Verified pass@1.
• Frontier trajectories required. Golden actions come from frontier-model trajectories, so the probes measure support for one agent's path. Only pass@K credits alternative correct solutions.
• Verifier quality matters. Distractor quality and golden-action validity both depend on the benchmark's tests.
• Truncated page. The fetched text ends partway through the Patch MCQ section. The following are not visible here:
• option count k
• pass@K sampling details
• compute cost
• per-model results tables
• the authors' own limitations section