AgentSWE: Can Coding Agents Build the Agent You Actually Want?

What it claims

AgentSWE tests whether coding agents can build agents that actually satisfy a user's requirement, rather than ones that only appear to. The authors say no existing benchmark tests this. Across the coding agents, models and harnesses they evaluated, the gap between what a builder demonstrates and what it delivers is "the norm, not the corner case." The abstract gives no scores, and it does not name the agents or models tested.

How it works

Each of the 25 tasks is framed as a commission rather than an exam. The builder sees only a handful of illustrative cases. Acceptance runs the frozen submission on held-out cases designed to span the full requirement. Grading differs by stage:

• Creation (a complete agent from a natural-language requirement) and Editing (changes to production agent codebases) are scored against rubrics derived from the requirement, with hard pass/fail gates.
• Optimization is scored against the objective of the source benchmark the task comes from, not against a custom rubric.

The split between visible and held-out cases is the core design choice. It is what exposes overfitting to the visible cases, one of the failure modes the paper reports. The paper also analyzes failure trajectories, and the authors present the failure modes it finds as signals that could be used to train coding agents.

How to use it

The page offers a CLI route to pull the paper into an agent:

hf papers read 2610.13361

If you don't have the latest CLI:

curl -LsSf https://hf.co/cli/install.sh | bash

No model or dataset on Hugging Face links to the paper, and one Space cites it. The page does not say where the benchmark tasks or harness can be obtained.

Limits

• The page contains only the abstract. It reports no per-agent results, task breakdowns, model names, harness details or compute figures.
• No dataset release is linked, so check whether the 25 tasks and the held-out cases are public before planning to run the benchmark.
• With 25 tasks across three stages, each stage has a small sample. The abstract does not say how the tasks are split between stages.
• The listed failure modes are introduced with "among them," so the list is not exhaustive.