Skip to content

Back to the catalog

Sharpening Tax in Post-Training
arxiv.orgpaper

Sharpening Tax in Post-Training

This paper studies the sharpening hypothesis of RL post-training on LLMs: that it mainly sharpens existing base-model behaviors, boosting pass@1 at the cost of solution coverage (pass@K), and shows this cost extends to agentic, multi-turn tool-use tasks, not just math/coding. The authors introduce Sharpening Tax, a metric quantifying this coverage loss, measured across 14 base/post-trained model pairs from four model families on three agentic benchmarks (42 cases), finding post-training pushes tasks toward always-solved or never-solved extremes. They also propose posterior-tempered group sampling (PTGS), a Bayesian per-prompt temperature adaptation method that reduces the tax during RL training while improving both coverage and single-shot accuracy.

TL;DR A new metric, Sharpening Tax, shows RL post-training sacrifices agentic solution coverage for single-shot accuracy, and a proposed fix (PTGS) mitigates it.
Open original ↗
Document

What it claims

On multi-turn tool-use tasks, pre-trained base models with a light, generic harness reach higher solution coverage than their instruction/RL-tuned counterparts once enough rollouts are sampled. Post-trained models still win at small K. On WebShop with gemma-4-31B, the base model exceeds 85% pass@128 against 56% for the post-trained model. The authors explain this as bimodalization: post-training moves tasks out of the "sometimes solved" middle and into the "always solved" and "never solved" extremes. Extra samples cannot help with either extreme. How much this costs depends on model scale and on the sampling budget you have.

How it works

  • Metrics. For each task they compute three unbiased estimators over n rollouts with c successes:
  • pass@1 measures accuracy.
  • pass@k measures coverage: the probability that at least one of k rollouts succeeds.
  • pass^k measures consistency: the probability that all k rollouts succeed.

Sharpening Tax reduces the base-vs-post difference in test-time scalability to one scalar. The fetched text does not include its formula.

  • Harness. Base models are prompted through a dataset-independent, model-agnostic scaffold: a simple system prompt plus relaxed tool-call parsing. Post-trained models use each dataset's default scaffolding, because the same harness hurts them.
  • Setup. There are 14 open checkpoint pairs from the Gemma-4, Ministral-3, Qwen2.5 and Qwen3.5 families, ranging from 3B to 35B effective parameters. Benchmarks are the BFCL v4 multi-turn base split, WebShop and ACEBench. The authors chose these because intermediate actions and state are checked, which means a lucky final answer cannot inflate pass@K. Thinking mode is turned off for post-trained models; Figure 10 ablates it.
  • PTGS. During RL rollout generation, a Bayesian per-prompt difficulty estimate sets the sampling temperature. Hard prompts get a smoother (hotter) temperature and easy prompts a sharper one. It plugs into PPO and GRPO.

Evidence

Harness ablation (average over 8 base models, without → with harness):

Benchmarkpass@1pass@32
BFCL5.59 → 15.6315.19 → 49.13
WebShop6.32 → 9.7339.48 → 49.15
ACEBench44.24 → 42.6389.25 → 87.69

The harness matters where base models struggle and does little where they already cope.

Crossover budget k\* (where the base model overtakes the post-trained one, Gemma-4):

  • WebShop: k\* > 128 at 4B, then 16 (12B), 11 (26B-A4B) and about 3 (31B).
  • ACEBench: 123 (12B), 43 (26B-A4B), 49 (31B).
  • BFCL: 32 (31B).

Larger models cross over sooner. Sharpening mostly helps small models by raising the floor, and mostly hurts large models by lowering the ceiling.

Bimodalization (WebShop, gemma-4-31B, 128 rollouts):

CategoryBasePost-trained
Pass given compute87.6%30.0%
Always pass0%26.0%
Always fail12.4%44.0%

The tax is reported as prevalent across most of the 42 cases. The authors also say it can be estimated from a few rollouts and correlates with other metrics.

How to use it

The fetched text gives no install commands.

The practical rule the paper states: if you run a large model with an abundant sampling budget and a verifier, and you only need one good solution, the base model with a harness may serve you better than the instruct model. Examples are search-style discovery and open problems.

To measure the tax on your own model pair, a handful of rollouts per task is reportedly enough to predict the many-rollout result.

Limits

  • The fetched text cuts off in §3. The Sharpening Tax definition, the full 42-case results, PTGS numbers (tested in only two agentic environments), and the theory in §6 are not shown. Treat PTGS's claimed gains in both accuracy and coverage as unverified here.
  • The comparison is asymmetric: base models get the harness, post-trained models get default scaffolding.
  • Post-training recipes for these checkpoints are undisclosed. "RL post-trained" is used loosely and may include SFT and DPO.
  • Results are without thinking mode, apart from one ablation.
  • Coverage gains assume a reliable verifier to pick the successful rollout.

Techniques named here

  • Sharpening Tax
  • Posterior-tempered group sampling

Related in research