VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

What it claims

VibeTraining-Bench (arXiv, October 2026, Alibaba Token Hub with Zhejiang and Westlake universities) tests whether a research agent can improve a fixed agent system by editing its training data, trainer and inference harness together. The authors call this "cross-component co-design." They place it against single-component benchmarks: RSIBench-Data and Curation-Bench for data, AI4AI-Bench for training algorithms, and HarnessOpt-Bench and Evo-Bench for harnesses. They also compare it with PostTrainBench and Agent2 RL-Bench, which cover data and training but leave the harness out. Their trajectory analysis finds that agents mostly do lightweight work, such as data filtering and hyperparameter tuning. Agents rarely change the training algorithm itself or make coordinated changes across all three components.

How it works

Each task gives the agent a base model, open-source training data, a trainer (algorithm code plus initial scripts), an inference harness and a fixed budget. The agent asks an isolated grader for validation runs. The grader returns a score and per-task execution trajectories. Candidate models can be newly trained, merged from existing models, or reused from an earlier iteration. At the end the agent submits one model and harness pair. The grader scores it once on a hidden test split, with no feedback, and the metric is the test-score gain over the initial system.

flowchart LR
  A[Initial model, data, trainer, harness] --> B[Research agent edits D / T / H]
  B -->|request validation| G[Isolated grader container]
  G -->|score + trajectories| B
  B -->|budget ends: submit once| T[Hidden test set]
  T --> S[Final score]

Reward hacking is blocked in three ways. Test data is never visible during research. Research and grading run in separate containers, so the agent cannot touch grader code or evaluation data. The research sandbox has no external network access. The authors cite PostTrainBench, where agents trained on test data, as the motivation.

Only two of the five tasks expose the harness. The other three fix their execution adapters or decoding settings.

Evidence

Six research-agent configurations were tested:

• GPT-5.6-Sol and GPT-6-Astra on Codex at xhigh reasoning effort.
• Kimi-K3, Qwen-3.8-Max, DeepSeek-V4-Pro and GLM-5.2 on Claude Code.

Every run gets eight NVIDIA A100 GPUs and starts from Qwen3.5-9B-Base with the same initial data. On SWE-bench Verified, GPT-6-Astra changed both training and the harness and raised the target system from 32.9% to 53.7%. The human post-trained instruct model scores 55.4%.

How to use it

Code and data are at https://github.com/D2I-ai/VibeTrainingBench. The page gives no install commands or licence. Reproducing one run needs eight A100s for 16 to 24 hours per task, plus API access for the research agent.

Limits

The fetched text stops at the experimental setup. Per-task results for the other four tasks and the other five agents, the trajectory statistics and the component ablations are not shown. Only the SWE-bench Verified figure is available. All runs use a single 9B base model. The authors state that agents do not yet achieve cumulative gains through iterative co-design.