Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

What it claims

Writing file-level documentation for a coding agent is not worth it when the agent can already read the source. The authors built a roundtrip benchmark and used it to optimize a description-writing prompt. With the source withheld, an optimized description raises issue resolution from a mean test-pass fraction of 0.08 to 0.71. With the source present, which is the normal way agents run, neither static compact docs nor retrieved past-task context beats giving the agent the issue alone. The authors report that longer documentation can make results slightly worse. A positive control shows the evaluation can detect a real gain when one exists.

Before running the large-scale test, the paper establishes two things in controlled settings:

• Documentation helps only when it carries something the code does not expose, such as intent or a contract the code does not enforce.
• Completeness drives value, not length.

How it works

The roundtrip benchmark has three stages:

1. Describe: a model writes a natural-language description of one source file. The original tests are hidden from it.
2. Regenerate: a separate model rebuilds the file from three inputs: the description, the package scaffold with the target file removed, and a contract file naming the public symbols. It never sees the original source.
3. Evaluate: the rebuilt file runs against the original tests.

flowchart LR
  S[Source file] --> D[Describe]
  D --> R[Regenerate]
  SC[Scaffold + symbol contract] --> R
  R --> E[Evaluate]
  T[Original tests<br/>hidden until here] --> E

Compactness is measured as density: tokens in the description divided by tokens in the code, using the same tokenizer. The roundtrip score is then used as the optimization signal for the prompt that writes descriptions. The fetched text does not include details of that optimization loop.

Evidence

Controlled experiments (Table 2). A weak agent (Flash-lite) used descriptions written by Pro:

• Graded facts: descriptions carrying 1, 3 or 5 of 5 key–value facts produced exactly 1, 3 and 5 correct answers.
• Length: a 38-word and a 664-word description with the same facts both scored 5/5. A harder variant showed no length effect across several models. It used 28 settings (10 checked) with confusable names such as max_retries and connection_max_retries, embedded in filler prose.

Benchmark fixtures.

• Source: SWE-bench Verified instances whose gold patch touches exactly one non-test file. 429 of 2,294 pass that filter.
• Restricting to sympy and to instances buildable with a current Python toolchain leaves 11 fixtures.
• Setup: Flash (gemini-3.5-flash) runs both describe and regenerate at temperature 0, three runs per fixture.

Benchmark results.

• Mean fidelity is 0.32 ± 0.35 and mean density is 0.69 ± 0.48.
• contains (48 lines) scores 1.00, but its description is longer than the code (density 1.57).
• Four fixtures score 0.00: symbol, pycode, gamma_matrices and rings (2,470 lines).
• Variance is high. unitsystem_v2 scores 0.61 ± 0.54.

How to use it

The paper links its code and data at https://github.com/haw-ai-i/roundtrip. The fetched text gives no install commands, licence or optimized prompt text. It says the prompts are in Appendix C of the supplementary material.

Practical takeaways:

• If your agent can load the file, adding generated file summaries is unlikely to help.
• If it cannot, a complete description is a strong substitute for the source.
• Hand-written contracts that the code does not enforce are where documentation pays off.

Limits

• The fetched text cuts off at Table 3. The optimizer's procedure and results ("full fidelity, generalizes to unseen files") are not in it. Neither are the downstream tables (two model families, ten repos, SWE-ContextBench, Table 8) beyond the 0.08→0.71 figure and the null result.
• The benchmark is small and narrow: 11 fixtures, all from sympy.
• The hosted API is not deterministic even at temperature 0, so scores are means over 3 runs.
• Contamination control relies on Verified being a decontamination target, plus monitoring for verbatim recitation. On widely mirrored files, the model sometimes refuses to generate, and such files are excluded.
• Density figures are only valid for per-file regeneration. They are not comparable to whole-package regeneration.
• The controlled experiments use very small samples (3–5 attempts) and hand-built codebases.