When Agents Fail: Model or Harness--Who's to Blame?

What it claims

The October 2026 paper from Shanghai Artificial Intelligence Laboratory finds that the harness ranking depends on which model drives it. OpenCode leads on Cross-Model Fit with 87.5, ahead of Pi CLI (77.1), Codex CLI (73.6) and DeepAgents (64.2). That aggregate covers three models on six benchmarks. On the shared four-benchmark panel, DeepSeek V4 Flash Preview has the highest Mean Valid Completion Rate across harnesses.

Spending more tokens did not buy better outcomes. Low token use can reflect premature termination, and high use can include repeated failed actions. The lightest harness changes from one configuration to the next.

Task score and valid completion can favour different harnesses for the same model. Pair-specific VCR deviations change sign across benchmarks.

How it works

The harness is modelled as a stateful runtime with six interfaces:

• Observe converts raw feedback into observations.
• Context formats instructions, history, tools and memory.
• Control handles retries, recovery, delegation and stopping.
• Act maps model proposals to operations.
• Update persists memory and artifacts.
• Govern enforces permissions, budgets, validation and handoff.

A run is "clean" (U = 1) when it both terminates normally (N) and hands the evaluator a consumable result (V), whether or not the task is solved. VCR is the mean of U over tasks and repetitions.

Fit combines task score with clean completion:

1. Each attempt contributes its normalized score S in [0,1], partial credit included, only if U = 1. Invalid attempts count as zero.
2. J is the mean of U·S over all scheduled attempts.
3. J is divided by the best J any harness reached for that model and benchmark.
4. Model-Harness Fit averages that ratio equally over benchmarks.
5. Cross-Model Fit is the geometric mean of Fit across models, which penalizes a harness that does poorly with any single model.

A Fit of 80 means 80% of the best observed valid-completion score for that model. Outcome Win Rate and VCR Win Rate are pairwise comparisons within each model-benchmark block, with ties counted as 0.5.

All harnesses run through a common agent adapter that keeps each one's native execution loop. Tasks, tool access and resource limits are matched across runs. Two fixed native model-harness systems serve as whole-system references.

Evidence

The study has 111 evaluation settings and 52,168 trajectories, counting native references and stress conditions.

• Models (four core): DeepSeek V4 Flash, MiniMax M3, Kimi K2.7 and Qwen3.6 35B-A3B.
• Benchmarks (six): SWE-bench-45, Terminal-Bench 2, VitaBench, LOCA-Bench, ClawMark and AgentSkillOS. Together they cover context management, tool use, long-context planning, multimodal interaction and artifact handoff.
• Model-specific Fit panel: four models on the first four benchmarks.
• Cross-Model Fit and token panels: three models on all six benchmarks. The fetched text does not say which model is dropped.

Limits

The fetched text stops at the metric definitions in Section 4. Per-benchmark tables, the failure taxonomy and the separation of abnormal termination from invalid handoff (Section 6.2.2) are not shown. The token-load figures beyond the overview plot are also missing.

Fit is relative to the best harness in the tested panel, not an absolute score. Adding or removing a harness changes every number.

The authors state that OpenCode's lead is stable only on the evaluated panel. Model-specific preferences and the intermediate ranks depend on which models and benchmarks are included.

Each harness is compared with its bundled design choices intact, so the results do not attribute effects to individual harness components. The fetched text mentions no code, adapter or trajectory release.