Same Model, Different Harness: Different Coding-Agent Results

What it claims

Yuj is a coding-agent harness. With the model held fixed, switching on its "treatment" configuration makes the agent solve more tasks when the context window is tight. The treatment is three mechanisms bundled together:

• a deterministic "half-life" rule that truncates old tool results;
• a stall detector that issues fixed interventions;
• a few command safeguards.

The primary model is Qwen3.6-35B-A3B at four-bit Q4 K XL, served with llama-server from llama.cpp. When the window is wide, the gains mostly vanish. The author says the individual mechanisms are not novel. The contribution is the tested package plus the paired evidence.

How it works

Both arms keep the complete conversation in memory. Trace and transcript files also preserve it. Only the model-facing view changes.

Half-life view rule. Shortening starts once the estimated prompt reaches 50% of the configured window. "Age" means how many newer tool results have arrived since a given result.

A shortened result keeps its head and tail around an omission marker. The task text and the model's own replies are never shortened. The rule makes no model call and produces no summary.

Detector. Fixed rules run over the execution record. Examples are a repeated command that hits the same failure, or repeated reading without any edit. When a rule fires, the harness injects a fixed message, for example "Loop detected … Stop repeating … change approach." It then keeps watching.

Safeguards. The harness rewrites test commands into the expected form, blocks forbidden commands, and stops oversized setup output from flooding the context.

Control. The control sends the full chronological transcript each turn and ends the run when the transcript no longer fits.

flowchart LR
  R[Complete record] --> V[Build view: shorten old results if ≥50% full]
  V --> M[Model action]
  M --> S[Command safeguards] --> T[Tool run]
  T --> R
  R --> D{Stall pattern?}
  D -- yes --> I[Fixed intervention] --> V
  D -- no --> V

Evidence

• Control validity. On all 500 SWE-bench Verified tasks with GPT-5.5 via Codex CLI, untreated Yuj and mini-SWE-agent v2.2.8 agreed on 87.6% of task outcomes (Cohen's kappa 0.66).
• Tight window, Verified. On a 169-task cohort with a 20,480-token window and a 480 s attempt limit, treatment raised mean F2PF from 28% to 49% and full solves from 43 to 72.
• Other models. The frozen treatment, without retuning, raised both metrics for three more models. The fetched text does not name them.
• Pressure comparisons overall. F2PF rose in all three benchmarks. Full solves rose on Verified and Pro.
• Shortening activity (Table 1).

• Wide window. Verified and Pro outcomes were close between arms. FeatureBench kept a higher F2PF under treatment. On the wide-window Verified cohort, treatment sent fewer prompt tokens per turn.
• Statistics. Resolution uses an exact McNemar test. F2PF uses an exact sign test with ties dropped, so the p-values speak to direction, not size. Per-task numerators and denominators are published in task_outcomes.tsv.

Limits

• No ablation. The three mechanisms ship as one package, so the effect of any single rule is not identified.
• Tuning overlap. The half-life settings were chosen during earlier development, on runs from the same benchmark families. There was no task-disjoint tuning, so the results may be optimistic.
• Untested alternatives. The paper does not compare against model-written summaries or retrieval-based views.
• Package versions differ across benchmarks. Pro and FeatureBench used an earlier frozen package. Their 262,144-token rows are not exact replications. The FeatureBench wide-window row pairs an older treatment with a later, corrected control.
• Scoring rule. A task with no F2P denominator scores F2PF = 0.
• Single runs. Each task ran once per arm.
• Possible harms. The author notes that shortening can hide a useful detail, and that interventions can interrupt repetition that was actually productive.
• Scope of the gains. The improvements are tied to context pressure. Under a wide window, expect little change on Verified or Pro.