Harness Learning Enables Generalizable Test-Time Adaptation

What it claims

A small trained proposer model can learn to rewrite an agent's harness code from execution feedback, and that skill transfers to tasks it never saw. On Reasoning Gym, the average single revision on 21 unseen task families rises from 0.32 with the untrained base proposer to 0.62 after RL. That beats the 35B teacher whose revisions seeded SFT, which scores 0.56. In multi-hop QA, a proposer trained only on HotpotQA transfers to MuSiQue and 2WikiMultihopQA. At test time both the proposer and the solver stay frozen; only the harness code changes.

How it works

Each revision round works as follows:

1. The proposer receives the task description, the current harness h_{t-1}, and an execution report from running that harness on feedback questions.
2. It emits structured edits, each naming a span of existing code and its replacement. Edits that cannot be parsed or applied are marked invalid.
3. Candidates are scored on a separate scoring set, and a selection rule picks the next harness.

The selection rule differs by setting:

• Reasoning Gym: one candidate per round (G=1). The candidate is kept only if its development score beats the current harness.
• QA: eight candidates per round. Candidates that fail to parse or run score zero. The loop advances to the best candidate even if it is worse than its parent.

flowchart LR
  A[Task + harness + exec report] --> P[Proposer]
  P --> E[G code edits]
  E --> V{Parses & runs?}
  V -- no --> Z[Invalid / score 0]
  V -- yes --> S[Score on scoring set]
  S --> R[Reward → GRPO update<br/>training only]
  S --> Sel[Select next harness]
  Sel --> F[Run on fresh feedback questions]
  F --> A

Training updates only the proposer, using GRPO. The reward is the candidate harness's mean score on the scoring set plus an auxiliary reward for edit validity and successful execution. During training, scoring questions are disjoint from the questions used to build the report.

The training objective is a deliberate surrogate. The goal is the held-out score after T rounds, but each update credits only the candidate's immediate outcome. SFT initialization is optional. Its data consists of 35B-teacher revisions of the seed harness that improved on the seed and passed quality filters.

"Multistep RL" means training on successive revision states rather than single revisions:

• Reasoning Gym: states are refreshed between offline phases, continuing from the Single-step RL checkpoint.
• QA: revision runs are generated online, starting from the base model with no SFT.

Evidence

Reasoning Gym setup

• Qwen3.5-4B serves as both proposer and solver.
• Training used 21 families for SFT and 5 for RL; 21 further families are held out.
• The seed harness makes a single solver call.
• Single-step results average 8 proposals on format-varied questions. Oracle best-of-8 is reported separately.

Reasoning Gym results

• The 0.32 → 0.62 gain is on the mean over all proposals, so it does not depend on picking the best candidate.
• After RL, the average proposal beats the seed harness. Base and SFT proposals fall below the seed on average.
• Multistep test-time revision: 5 rounds, one proposal per round.

Multi-hop QA setup and results

• Qwen3-4B proposer with a Qwen3-8B solver.
• The task is improving an existing retrieval workflow, measured by exact match.
• Protocol: 80 independent proposals, plus 4 runs of 10 rounds with 8 proposals per round.
• A proposer trained only on seed-harness inputs still improved already-revised harnesses over successive rounds.

How to use it

The page gives no code release, weights, licence, or compute figures.

If you want to replicate the recipe, the concrete pieces are:

• a span-replacement edit format;
• disjoint question sets for feedback, scoring, and evaluation;
• a validity-shaped reward;
• one of the two selection policies above.

Note that test-time adaptation still requires graded development questions for each new task.

Limits

• Multistep training is inconsistent. The authors state that the benefit of training on revision sequences "var[ies] across settings." Single-step-trained proposers already continue improving over multiple rounds.
• The training objective is a proxy. It optimizes immediate revision gains, not final-harness quality.
• Scope is narrow. Only 4B/8B Qwen models and two domains (synthetic reasoning, multi-hop QA) are tested. No coding or tool-heavy agent benchmarks are included.
• Best-of-N numbers are oracles. They are selected on held-out scores and should not be read as achievable performance.
• Closely related concurrent work exists. Harness-R1 uses a similar SFT+RL recipe with a frozen solver. JIT-Agent and TTHE (which adapts using unlabeled traces) are also relevant. Compare against these before building.
• The fetched text is incomplete. It cuts off mid-results, so the QA numbers, the multistep comparisons, and the analysis of generated tools are not available here.