Skip to content

Back to the catalog

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
arxiv.orgpaper

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

This arXiv paper studies why adding procedural "skills" to LLM agents can hurt as well as help performance, running nearly 6,000 evaluation runs across two office automation benchmarks and three model harness stacks. It distinguishes "regressions" (tasks solved without skills but failed after adding them) from "residual failures," and identifies three regression causes: skill description osmosis, grounding displacement, and verification displacement. The authors argue procedural skills are overemphasized relative to grounding and verification, and that skill evaluation should decompose net effect into gains vs. regressions rather than reporting aggregate improvement alone.

TL;DR A large-scale study finds LLM agent "skills" often help by regressing less rather than gaining more, and pinpoints three failure modes (description osmosis, grounding displacement, verification displacement) that cause skills to backfire.
Open original ↗
Document

What it claims

The paper puts numbers on how often procedural skills break tasks LLM agents had already solved. Across 5,832 paired runs (486 tasks × 4 conditions × 3 stacks), it counts 553 gain transitions against 324 regression transitions — regressions offset 59% of gross gains, leaving a net of 229. The headline claim: the best libraries win by regressing less, not gaining more, and reliability depends more on grounding and verification than on which procedural skill you pick.

How it works

Four-stage pipeline per stack: baseline run → analyst extracts failure signals → meta-skill creator turns signals into skills → evaluation run. Crucially, the same analyst signals feed three different skill creators (Anthropic's measurement-driven eval loop, OpenAI's single-pass structural check, and the authors' "Ours" self-critique-with-reuse), so one signal set yields three libraries ranging 3 to 23 skills. Each skill has a description (always in system prompt) and a body (loads only on invocation); this split lets them isolate presence-only effects. Each task lands in one of four cells against the no-skill baseline: gain, regression, residual failure, retained. Net effect = gains − regressions, tested with exact two-sided McNemar on discordant pairs.

Evidence

  • Two benchmarks: OfficeQA-Pro (94 curated QA tasks over U.S. Treasury PDFs, correct within ~1%) and SpreadsheetBench (392 Excel-forum edit tasks, exact cell-value grading).
  • Three stacks: OpenCode·MiniMax-M2.7, Codex·GPT-5.4-mini, Claude Code·Sonnet-4.6.
  • Decomposition flips rankings: on Claude Code + OfficeQA-Pro, libraries gained 10/11/12 but regressed 2/4/7 — ranking by gains puts openai first; ranking by net puts anthropic first (+8).
  • Only 5 of 18 conditions reach p<.05 — all three libraries on Claude Code·Sonnet-4.6 + SpreadsheetBench (+43, +43, +47), plus a couple on the anthropic library. Most conditions are statistically null.
  • OfficeQA-Pro: 122 gained, 81 broken (66% cancelled). SpreadsheetBench: 431 gained, 243 broken (56% cancelled).

Limits

  • Grading artifact: the SpreadsheetBench engine can't recompute structured/table references or several modern Excel functions, so correct formulas score as failures; authors re-graded 226 treatment task-conditions in §5.3 and report both raw and corrected.
  • Harness and model are coupled — each stack is one unit, so effects can't be attributed to model vs. harness separately.
  • Each condition run once, no seed replication, so run-to-run variance isn't estimated (flagged for single-task flips).
  • Libraries aren't length-matched, so treatment-vs-treatment isn't a controlled token dose — used only for tracing individual outcomes.
  • Preprint; the fetched text cuts off mid-§4.3, so the mechanism-attribution results (§5) aren't in this excerpt.

One caveat worth flagging: the arXiv ID (2607.22520) and dates (Jul 2026), plus model names like GPT-5.4-mini and Sonnet 4.6, place this beyond my knowledge cutoff — treat the stack identities as reported by the page, not independently verified.

Techniques named here

  • Procedural skills for LLM agents
  • Regression
  • Residual failure
  • Skill description osmosis
  • Grounding displacement
  • Verification displacement

Related in prompting