Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

What it claims

Doc2Harness is presented as the first deterministic benchmark for long-horizon harness engineering: it measures how well a model can build the scaffolding around a coding agent, not how well an agent performs under a fixed harness. Across 19 frontier models, each with three independent builds scored on 328 test cases, GPT-6 Astra reaches a 53.05% pass rate and the weakest model reaches 27.44%, a gap of about 26 points. Even the best model fails nearly half the cases.

How it works

The benchmark is deterministic because a scripted mock model drives the harness in Stage 2. Pass/fail therefore does not depend on a live LLM's sampling. A second detail the summary omits is that positive controls are used to rule out vacuous passes, so a harness that does nothing cannot score on channels that happen to match an empty outcome.

flowchart LR
  A[Contract skeleton] --> B[Build model in fixed builder harness]
  D[Dev doc from 12 harnesses] --> B
  B --> C[Frozen candidate harness]
  C -->|new container per test| E[Run under scripted mock model]
  E --> F1[Model requests]
  E --> F2[Files on disk]
  E --> F3[Exit code]
  E --> F4[Command output]
  F1 & F2 & F3 & F4 --> G{Pass?}
  P[Positive controls] --> G

The construction container and the evaluation containers are separate. The built harness is frozen before testing, so it cannot adapt to the test environment.

How to use it

The page provides a way to load the paper into an agent:

hf papers read 2610.20344

To install the CLI:

curl -LsSf https://hf.co/cli/install.sh | bash

No models or datasets are linked to the paper on the page. The page does not give a code repository, test-case release or licence.

Limits

• The page contains only the abstract. Per-model results beyond the top and bottom scores, the variance across the three builds, and the breakdown of the 328 cases are not shown.
• The failure finding is stated only at a high level: failures cluster around how cross-cutting functionality is split across modules. No taxonomy or examples are given.
• Scoring uses a mock model, so the results measure conformance to the consolidated spec. They do not measure whether the built harness improves downstream task success with a real model.
• Before planning to run the benchmark yourself, check whether the benchmark artifacts are actually released.