Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents

What it claims

Choosing a single-step repair for a failed LLM agent using a substitute evaluator loses measurable deployment value. The baseline is selection with the deployed agent's own rollouts. On unscreened failure pools, greedy replay loses 0.08 to 0.13 in net success gain in ALFWorld and 0.06 to 0.13 in net reward in WebShop. At equal rollout budget, using Qwen2.5-14B rollouts to choose repairs for Qwen2.5-7B loses 0.25 in ALFWorld and 0.07 in WebShop. Twenty 14B rollouts per candidate pick repairs worth about one 7B rollout in ALFWorld and fewer than two in WebShop. Fresh rollouts show 7B and 14B prefer opposite repairs on selected disagreement cases. Llama-3.1-8B, in contrast, mostly fails to separate the repairs 7B needs. The scoring executor also shifts repair-location labels and changes which repair method a benchmark ranks first. A few deployed-agent rollouts per candidate recover most of the loss.

How it works

A repair replaces action a_t with a′. Its net gain under executor e is p_e(h_t, a′) − p_e(h_t, a_t), so repairs are not credited for successes the original action would also reach on a rerun. An "executor" is the weights plus prompt and history protocol, decoding parameters and invalid-action handling.

Failed trajectories and candidate sets are held fixed. Every candidate is labelled with rollouts of the deployed policy. Matched selection is cross-fitted: five of ten rollouts select, five evaluate, averaged over 50 splits. Selection loss is the expected difference in held-out deployment value between matched selection and the evaluator's choice. As a positive control, matched selection must beat random selection by at least 0.10, or the pool is marked uninformative. Tests were pre-specified in a dated analysis plan, with one-sided α = 0.05, Holm correction and 2,000 bootstrap replicates.

Evaluators fall into two groups:

• Target mismatch: greedy replay, temperature 0.3, a solvability oracle, a planner, other models' rollouts and an LLM judge (Qwen3.8-flash with reasoning).
• Budget mismatch: 1 to 5 own rollouts, and re-implemented verification rules: 2-of-3 acceptance (Ma et al., 2026), single-replay flip, earliest single success, and latest significant step.

Evidence

The environments are a 5×5 gridworld with exact solvability and ALFWorld with a Fast Downward planner and a 50-step budget (two-object tasks excluded). The third is WebShop, using the 100,000-product setup, 1,021 human instructions, BM25 and a 25-step budget. Reward is the primary WebShop metric.

Each unscreened pool holds the first 60 failures in collection order, for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct as deployed agents. Both run on vLLM at temperature 0.7 and top-p 0.95. Candidates are two proposals from each of three LLMs, a privileged action and a random action. In WebShop every clickable element is added, giving about 6 candidates per position in ALFWorld and 10 to 11 in WebShop.

Cross-family losses between 7B and Llama appear in both directions in both environments, at 0.07 to 0.17. On the screened pools (34 gridworld and 30 ALFWorld failures), Figure 2 shows larger losses in ALFWorld: 0.29 for 14B and 0.63 for Qwen3.8-flash rollouts. The paper reports that oracles, published verification rules and LLM judges show the same pattern there.

How to use it

The authors release code, trajectories, all rollout outcomes and the analysis plan at https://github.com/lyj20071013/repairs-for-whom. The page states no licence. The practical recommendation is to verify candidate repairs with a few rollouts of the executor that will actually run them, using an identical prompt and decoding setup. Repair labels should also record which executor produced them.

Limits

Only single-step repairs are studied, and the deployed agents are 7B to 8B models. ALFWorld has no irreversible actions, so every failure there is an exhausted budget. The unscreened pools still exclude ALFWorld's two-object tasks and place repairs only where at least five steps of slack remain. The published verification rules are re-implementations of the verification step only, not full reproductions. The fetched text ends partway through the setup section. Sections 4 to 7 are known only from the introduction's summary, including when cheap verification suffices and when the candidate set is the bottleneck.