ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement

What it claims

An agent harness takes Qwen3.5-4B (Base or Instruct) plus a target benchmark and runs autonomous post-training rounds. It is limited to 24 hours or three rounds on resources equivalent to eight H20 GPUs. Its headline comparison is against Codex and Claude Code given the same budget, and it beats both on all 11 Base-model tasks. The largest gain is +41.96 points on LiveCodeBench v6 (19.64 → 61.60), where Claude Code reached 42.63.

How it works

Each round is one post-training trial, framed as one optimizer step.

• Error attribution. The agent evaluates the current checkpoint on the task's validation split. It reads both aggregate or subtask metrics and individual failed predictions, then picks one focal problem by assessed impact. The authors restrict each round to one problem because, when shown several, the agent drifted toward easier secondary issues.
• Update direction. The agent must research several candidate strategies from papers, code and datasets before choosing. Each candidate is recorded with its source, applicability, reported gain and a three-level reproduction difficulty:
• Low: released, training-ready data.
• Medium: needs processing, generation or distillation.
• High: complex procedure or incomplete resources.

The candidate pool combines three sources: fresh research, unselected candidates and outcomes from earlier rounds, and a library of validated strategies and scripts. Resources are verified before execution.

• Executable update. The agent builds training data and a config. Batch size and parallelism are adapted to the hardware. Before the full run, it runs preflight checks: data preparation on a few real samples (content, format, template) and a short training run.

flowchart TD
  E[Evaluate on validation split] --> A[Metric + case-level attribution]
  A --> F[Single focal problem]
  F --> R[Research candidates]
  H[Trial history + asset library] --> R
  R --> S[Select by gain / applicability / difficulty]
  S --> P[Preflight: sample data + short run]
  P --> T[Full post-training]
  T --> C{Gain on re-eval?}
  C -- yes --> Acc[Accept candidate, store scripts]
  C -- no --> K[Keep parent, record failure]
  Acc --> H
  K --> H
  Acc --> E
  K --> E

Evidence

All numbers below are for Qwen3.5-4B-Base under the same budget (scores %). The columns are initial checkpoint / Codex / Claude Code / ImproveAnyTask:

*BFCL excludes Web Search, and the remaining weights are renormalized.

The mean Base gain is 18.29 points. The mean Instruct gain is 11.97 points, but the per-task Instruct table (Table 1) is not in the fetched text.

Limits

• Some margins are small. On IFEval the harness essentially ties Codex (74.50 vs 74.29). On AIME 2025 it gains 4 points.
• Narrow model coverage. Only one model family and size (4B) is reported.
• Selection uses the validation split. Checkpoints are chosen on the task's validation split. Check that final numbers come from a separate test split before trusting the gains.
• Baselines underspecified. The fetched text does not state which models or versions powered the Codex and Claude Code baselines, nor which model drives the harness itself.
• Missing sections. The page text is truncated mid-method. Timing and stopping rules (Section 4.1), the error-attribution evaluation (Section 4.4), ablations and any authors' limitations section are not available here.
• No release details. No code release, licence or run instructions appear in the fetched text.