Harness Engineering for Self-Improvement

What it claims

The near-term route to recursive self-improvement is not a model rewriting its own weights. It is making the harness itself the optimization target. The object being optimized moves through prompts → structured context → workflow → harness code → optimizer code. Weng adds a caveat that runs through the post: harness gains depend on base-model capability, so "intelligence is still the core". She also predicts that many harness tricks will be absorbed into model behaviour, much as prompt tricks faded once instruction tuning improved. The interface to tools and context will remain.

How it works

The post pulls out concrete design rules that the summary only names:

• ACE: the curator never rewrites the whole prompt. It emits (identifier, description) bullets that are merged by deterministic logic and periodically deduplicated. This is the stated defence against context collapse and brevity bias.
• MCE: separates the mechanism from the content. A "skill" is static components (prompts, knowledge bases, code) plus dynamic operators (search, filter, format), stored as files such as skill.md. An outer loop does agentic crossover over past skills, scored on validation. An inner loop fits the context on training data.
• Meta-Harness: the proposer is a coding agent that reads the full execution history with grep and cat rather than loading it into context. It outputs a Pareto set of harness candidates.
• Self-Harness: a gated loop. 1. Cluster failures by root cause, not by surface verifier outcome. A timeout can hide different underlying causes. 2. Propose narrow edits that target recurrent, addressable failures. 3. Accept an edit only if it shows no regression on both the held-in and held-out splits.
• AHE:
• The harness is split into 7 file-backed components: system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration and long-term memory.
• The runs directory, tracer, verifier and LLM config are read-only. This blocks hacks such as disabling the verifier, swapping the model or raising the reasoning budget.
• Every edit carries a falsifiable prediction listing the expected fixes and at-risk regressions.
• DGM: the parent is sampled in proportion to its score and inversely to its child count. Edits use only two tools, bash and an editor (view/create/edit).
• ShinkaEvolve: rejects candidates whose code-embedding cosine similarity to the population is too high.

flowchart LR
  A[Run harness h_t] --> B[Mine failures by root cause]
  B --> C[Propose bounded edits]
  C --> D{No regression on held-in AND held-out?}
  D -- yes --> E[Merge into h_t+1]
  D -- no --> F[Log; harness unchanged]
  E --> A
  V[Verifier, permissions: outside loop] -.-> D

Evidence

• STOP: improved mean downstream performance across iterations with GPT-4, but degraded with GPT-3.5 and Mixtral.
• Lin et al. 2026: models from Qwen3.5-9B to Claude Opus 4.6 showed similar ability to write harness edits. A 9B proposer produced a skill procedurally isomorphic to Opus's. Benefiting from a harness is a different skill: it needs timely tool and skill invocation and long-horizon instruction following.
• AHE on Terminal-Bench-2:
• Beat the human-designed harnesses OpenCode, Terminus-2 and Codex, except on the Hard tier.
• Beat some self-evolve baselines, ACE and TF-GRPO.
• The frozen harness transferred to SWE-bench Verified.
• Self-Harness: learned model-specific instructions for MiniMax M2.5, Qwen3.5-35B-A3B and GLM-5 on Terminal-Bench-2, improving held-out pass rate. No figures are given.
• DGM: with Claude 3.5 Sonnet as a fixed model, it went from 20% to 50% on SWE-bench Verified and from 14.2% to 30.7% on Polyglot.
• AFlow: "decent improvement" over manual workflows and ADAS on QA, code and math. No numbers are given.
• Appendix benchmarks:
• RE-Bench: agents scored 4× higher than humans at a 2-hour budget, but humans won at 8 and 32 hours.
• MLE-bench: o1-preview with AIDE reached bronze in 16.9% of 75 competitions.
• PaperBench: best was about 21% (Claude 3.5 Sonnet).
• CORE-Bench: 21% on the hardest tier.

Limits

• SIA (joint harness and weight updates): Weng calls the evidence provisional. The task agent (gpt-oss-120b) is much weaker than the meta and feedback agents (Claude Sonnet 4.6), and the baselines are too weak to compare against.
• Autodata: trains only the weak solver, so Weng reads it as indirect distillation more than RSI.
• Evolutionary methods: work when fitness is cheap and objective, such as kernels, matrix multiplication or scheduling. They struggle when evaluation is slow or fuzzy. Compute efficiency is also flagged.
• Self-editing harnesses: break abstraction boundaries. Permissions and evaluators must sit outside the editable surface, and reward hacking is not solved.
• Trehan & Chopra: of the ideas tried, only 4 reached the full pipeline and only 1 became a paper. Failure modes they recorded:
• stale library defaults
• implementation drift toward simpler methods
• context loss
• declaring success on noise
• missing tacit domain knowledge
• weak scientific taste
• Unsolved by any surveyed method: sandbox RLVR does not capture repo maintainability, backward compatibility or migration cost.
• This is a survey. Most results are one-line restatements of other papers, so check the cited arXiv IDs before relying on any number.