Same Model, Different Harness: Different Coding-Agent Results

What it claims

Changing only the harness configuration lets the same weights do more under context pressure. The weights tested are mainly Qwen3.6-35B-A3B, 4-bit Q4_K_XL, served by llama.cpp's llama-server. The gain comes from a fixed package of mechanical rules with no extra model calls: tool-result truncation by age, a stall/loop detector with canned interventions, and command safeguards. The paper claims no novelty for the individual mechanisms. Its contribution is the tested package and the paired evidence. When the context window is wide, the advantage mostly disappears on Verified and Pro.

How it works

The control feeds the full chronological transcript and stops the run when it no longer fits. The treatment keeps the full conversation in memory and on disk, but rebuilds what the model sees on each step:

• Half-life truncation. It activates once the estimated prompt reaches 50% of the window.
• The newest 4 tool results stay verbatim.
• Results of age 4–7 are capped at 4,096 characters, 8–15 at 2,048, 16–31 at 1,024, 32–63 at 512, and anything older at 256.
• "Age" means the number of newer tool results that have arrived since.
• A truncated result keeps its head and tail around an omission marker.
• Task text and the model's own replies are never shortened.
• There is no summarization, field extraction or retrieval.
• Detector. It checks fixed facts in the execution record, such as:
• the same command producing the same failure;
• repeated reading with no edit, e.g. the same arguments reissued after 7 turns with no source write.

When a pattern fires, the harness injects a fixed intervention ("Loop detected … change approach") and keeps watching.

• Command safeguards. These normalize test commands to the expected form, block forbidden commands before they run, and keep oversized setup output out of the context.

flowchart LR
  R[Complete record] --> V[Build view: truncate old results if ≥50% full]
  V --> M[Model action]
  M --> S[Command safeguards]
  S --> T[Tool run]
  T --> R
  R --> D{Detector: stall pattern?}
  D -- yes --> I[Fixed intervention] --> V
  D -- no --> V

Evidence

• Credibility of the control. On all 500 SWE-bench Verified tasks with GPT-5.5 via Codex CLI, untreated Yuj and mini-SWE-agent v2.2.8 agreed on 87.6% of task outcomes (Cohen's κ = 0.66).
• Headline result. On a 169-task Verified cohort with a 20,480-token window and a 480-second cap, mean F2PF rose from 28% to 49% and resolved tasks from 43 to 72. Three additional models also improved on both measures without retuning.
• How often truncation fired. Counts are runs with at least one shortening event; turn and reduction figures are medians.

• Wide window (Qwen3.6).
• Verified and Pro outcomes were close between arms.
• FeatureBench kept a higher mean F2PF under treatment.
• On Verified, the treatment also served fewer prompt tokens per turn.
• Statistics.
• Resolution is compared with an exact McNemar test.
• F2PF is compared with an exact sign test on per-task differences, ties dropped. This tests direction only.
• A record with no F2P denominator is scored 0.

How to use it

If you compare coding agents, report the harness configuration alongside the model. The paper's framing is that model plus harness is the solver.

If you run small-context or local models, the truncation schedule and the stall checks are cheap and deterministic, and can be reproduced from the values given above.

Per-task numerators and denominators are published in task_outcomes.tsv.

Limits

• No ablation. The package is tested as a whole, so the contribution of truncation, detector and safeguards individually is unknown.
• Tuning overlap. The truncation thresholds were set during development using runs from the same benchmark families. They were not chosen by a held-out, task-disjoint study.
• No alternatives compared. Truncation is not compared against model-written summaries or retrieval-based views.
• Mixed package versions. Pro and FeatureBench used an earlier frozen package. Their 262,144-token rows come from earlier campaigns. The FeatureBench wide-window row pairs an earlier treatment with a later, corrected control.
• Test conditions. Each arm runs once per task, and the main model is quantized.
• Truncated source. The fetched text stops partway through the statistics section. It does not name the three additional models or give their numbers, or report p-values for each comparison.