FeatureArc: Benchmarking Coding Agents on Evolving Features

What it claims

FeatureArc tests whether coding agents can keep a feature correct while they extend their own earlier code. The authors are from Tencent's Hunyuan team and CUHK, and the paper was posted October 2026. The setting is called "L4" delegation: the user gives only the next specification and never reports whether earlier stages failed. L1 to L3 cover settings where the user reports failures or explains fixes; Tencent's internal multi-stage evaluations covered only those levels before this work. The strongest agent named, Claude Opus 5.5, passes 83.6% of first stages but completes only 49.3% of whole features. The pipeline is also used internally to evaluate Hunyuan 4 Preview, a pre-release model.

How it works

Each task is an ordered chain of stages from one user-visible feature in one repository. Each stage comes with a requirements document and an interface contract (names, signatures, constants, error messages). The agent works in one persistent session and workspace. After each stage, a hidden "stage gate" runs F2P and P2P tests, and both must pass. The result is never shown to the agent, and failed work is not rolled back. After the last stage, chain-level F2P and P2P suites test the complete feature on the final workspace.

flowchart TD
  S[Next stage spec] --> A[Agent edits own workspace]
  A --> G{Hidden stage gate: F2P + P2P}
  G -->|pass| N[Carry code forward]
  G -->|fail: no feedback, no rollback| N
  N -->|more stages| S
  N -->|last stage done| C[Chain gate on final code]

Construction has five steps:

1. Select repositories with at least 30 stars, permissive licences, offline Linux builds, and mostly Python, TypeScript, Rust or Go code. Build a module map from imports so a feature can be tracked through renames.
2. Mine chains from an evidence graph of commits using four searches: beam, anchor-guided, endpoint bridging and community-guided. An LLM review agent then checks each chain against code, tests and diffs.
3. Remove the stage patches from the final checkpoint to get the start state S0, then reapply them to get S1 to Sn.
4. Build one offline container per chain. F2P tests must fail before a stage and pass after it; P2P tests must pass on both.
5. Generate the specification from the stage patch and its F2P tests. Documents that reveal later implementations are reverted.

An earlier internal attempt split single feature requests into stages. It produced stages that behaved as independent tasks, which is why the stages are now mined from history.

Evidence

From a pool of 100 stratified candidates (25 per language), seven were excluded after multi-model trajectory screening. Five of those had chain suites requiring unspecified behaviour. The final set is 73 tasks, 263 stages and 53 repositories, with 3.60 stages on average. A later stage shares on average 54.3% of its changed files with earlier stages (median 59.1%). Median reference changes per stage are 426 source lines and 416 test lines.

Across 10 agents, after a failed stage the next stage's F2P tests pass in 22.6% of cases, against 70.2% after a passed stage. Regression of earlier behaviour is rare: it affects only 3.3% of features whose new behaviour works. In 94.9% of stages following a failure, the agent never notices the earlier error, and agents' self-tests rarely revisit earlier stages. The dominant failure is building on broken earlier code, not breaking working code.

Limits

The authors chose a 73-task subset because of the cost of sequential multi-stage runs, so per-agent numbers rest on small samples. The fetched text breaks off partway through the benchmark-characteristics section. The per-agent result tables, the list of the 10 agents beyond Claude Opus 5.5, and the Hunyuan findings (Section VI) are not visible. The visible text does not state whether the tasks, containers or pipeline are released, or under what licence. Only L4 is evaluated, with no feedback between stages.