Aditya (@adityawaslost) on X

What it claims

Resolve is a 16-item Kaggle benchmark of single-function Python edits. An item scores only if the edit adds the requested behavior and keeps old behavior the prompt never restated. The author's main point is that one flat score mixes three different failures, and the per-item breakdown shows they do not move together. The benchmark is public on Kaggle as version 1 under Apache 2.0. A full write-up with charts is on DEV.

How it works

• Prompt: the original function, one requested change, and two sample inputs. Only the returned function is kept and everything else in the output is discarded.
• Grading: hidden tests run in two lists, fail-to-pass and pass-to-pass. Every test in both lists must pass. A timeout, crash, missing function or empty suite counts as a failure, and a half-fix scores 0 on that item.
• resolve-plus: applies the same rule plus a hidden edge suite. It is a fresh generation of the 16 items, not a regrade of the first one. Because of that, a model can score higher on resolve-plus than on resolve.
• Differences from SWE-bench: Resolve keeps SWE-bench's "resolved" rule but drops scraped repos, gold patches and model-written tests. The functions are the author's own.

Evidence

Every result is one sample per task over 16 items, so each item is worth 0.0625.

Diagnosed failures from the first four-model pass:

• ring_push: Haiku 4.5, Gemma 4 31B, GPT-5.4 mini and Gemini 3.7 Flash all passed fail-to-pass and failed pass-to-pass, on both tasks. That is the same broken unstated invariant in 8 of 8 runs.
• GPT-5.4 mini: its resolve-plus drop came from the edge suite on truncate.
• Gemini 3.7 Flash: its drop came from sampling variance. collapse_name passed on the first generation and failed on the second.

Price vs score: prices are list output prices per million tokens from a rate card dated 8 October 2026. They are not measured spend.

How to use it

The page gives no commands. The benchmark and its two task pages are on Kaggle. The task pages list every run, including unfinished ones. The method is worth reusing for your own edit evals: return only the function, require both test lists to pass, and run the edge suite as a separate sample so variance can be told apart from edge-case misses.

Limits

• Sample size: with 16 items and one sample per task, most gaps between models are 1–2 items. The collapse_name flip shows that a single resample can move a score by one item. Treat the ranking as coarse.
• Prices: no token or dollar logging was done, so the cost chart reflects list price only. It excludes open models, Gemini 3.6 Flash and Gemini 3.7 Flash.
• Missing models: Grok 4.5 and 4.6 were not run because the proxy returned "model not found". Qwen 3 Next 80B Thinking timed out three times and is excluded.
• Scope: the items are author-written Python functions, not real repositories, so the results do not cover multi-file edits.