Between Spoilers and Guesswork: Avoiding Over-Specification in Coding Benchmarks

What it claims

The paper, by authors from IBM and Rensselaer Polytechnic Institute (October 2026), proposes two policies for building identifier hints in repository-level coding benchmarks. DefUse chooses which identifiers to disclose. Declaration chooses how much to reveal about each one. Together they aim to remove name-mismatch failures without leaking the reference solution. The hint pipeline and the generated hints are already open source at https://github.com/IBM/swe-hints/, so this is a current release and not only a plan. The authors report that higher solve rates can come with more disclosure of implementation choices. On that basis they argue that hint schemes need to be judged on more than solve rate.

How it works

Existing schemes are compared along two axes. Selection: SWE-bench Pro uses manual choice by human engineers, FEA-Bench takes all new definitions, and Doc2Feat takes identifiers that appear in both the code and test patches. Projection: the options are name, type, path, signature, semantics, behaviour and docs. Doc2Feat's intersection rule still picks up field accesses and general imports, because it does not separate where an identifier is defined from where it is used.

DefUse keeps only the newly introduced names that cross the code–test boundary:

flowchart LR
  A[def in code patch] --> X1[∩]
  B[use in test patch] --> X1
  C[def in test patch] --> X2[∩]
  D[use in code patch] --> X2
  X1 --> U[∪ = DefUse hints]
  X2 --> U

The authors argue this set is both necessary and sufficient. Each selected name can cause a false failure if it is hidden. No discarded name can affect a test outcome.

Declaration reveals only what a stub would show: the name, its kind, the declaring file path, whether that file is new, and the signature for callables. It omits LLM-written summaries and mechanism descriptions. One example comes from an Ansible SWE-bench Pro task. Declaration gives get_best_parsable_locale(module, preferences=None) in the new file lib/ansible/module_utils/common/locale.py. The full hint for that task also names the locale -a command, the preferred locales and the exception to raise. It is thirteen times the length of the Declaration hint.

Evidence

The study covers five benchmarks: Doc2Feat-bench, FEA-Bench, SWE-bench Pro, SWE-rebench (all Python) and Multi-SWE-bench Java. It uses three language models, which the excerpt does not name. The Doc2Feat and FEA-Bench pipelines were reimplemented and reproduced their released hints, with minor bug fixes. The SWE-bench Pro hint pipeline was rebuilt using Claude Haiku 4.5.

Table 2 shows that DefUse is often empty, because many tasks' tests reference no new identifier:

The attribute ablation (Section 4.5) and the LLM judge (Section 4.6) are reported qualitatively in the fetched text. Structural attributes raise resolve rate with no solution disclosure. The further gain from semantic attributes comes with disclosed solution content.

Limits

The fetched text ends at Table 2, so the resolve-rate numbers for RQ1 to RQ4 are not available here. Evaluation subsets are small: 26 to 142 instances per benchmark. Disclosure is measured by an LLM judge, not by humans. The SWE-bench Pro hints are the authors' regeneration with Claude Haiku 4.5, not the original human-written hints. DefUse addresses only identifier-naming guesswork. It gives no hint at all on tasks whose tests use no new identifiers.