SAGE: A Statistical Acceptance Gate for Self-Evolving Agents

What it claims

SAGE replaces the acceptance gate in skill-document self-evolution with a three-condition rule. The baseline, SkillOpt's gate, accepts any edit that strictly raises mean validation accuracy. SAGE accepts an edit only when all three conditions hold:

• the asymmetrically weighted net gain is positive,
• a one-sided exact binomial test on the discordant items rejects at level α,
• the edit breaks no more than a fraction τ of already-solved items.

It is a strict refinement: disabling both components recovers the baseline gate exactly, so SAGE only ever commits a subset of the edits the baseline would commit. The paper also quantifies the problem under the naive gate. Regression rates range from 2.5% on ALFWorld and 4.8% on SearchQA to 42.8% on OfficeQA.

How it works

For each candidate edit, the gate runs the incumbent skill and the edited skill on the same validation items, with a binary verifier. It counts two kinds of discordant item:

• w (wins): the edit solves an item the incumbent missed.
• ℓ (regressions): the edit breaks an item the incumbent solved.

Items that both solve or both fail are ignored. From these counts it computes:

• Discounted gain: Δλ = w − λℓ, with λ ≥ 1, so a regression costs at least as much as a fix earns.
• Regression fraction: ρ = ℓ / n_S, where n_S is the number of items the incumbent solves.
• Significance: with m = w + ℓ, p = Pr[Bin(m, λ/(1+λ)) ≥ w]. This tests whether wins dominate losses beyond the break-even rate λ/(1+λ). At λ = 1 it reduces to the one-sided exact McNemar test.

What decides acceptance is the mix of wins and losses, not the net count. At λ = 1, 5 wins to 3 losses is not significant, but 8 wins to 0 losses is. If m = 0, p = 1 and the gate abstains.

flowchart TD
  E[Candidate edit] --> P[Run incumbent and candidate on same items]
  P --> L[Count wins w and regressions ℓ]
  L --> C{"Δλ>0 AND p<α AND ρ≤τ?"}
  C -- yes --> A[Commit edit]
  C -- no --> K[Abstain, keep incumbent]

The authors explain why they test paired counts rather than shrinking aggregate scores. All edits are scored on a shared validation set, so their scores are correlated through item difficulty. That correlation mis-estimates the shrinkage factor, and lower error in each score does not imply a more accurate ranking of edits. Pairing holds item difficulty fixed within each comparison.

The gate sits on top of SkillOpt's optimizer, which proposes a mini-batch of b candidate edits per step. SkillOpt uses a textual learning rate, a rejected-edit buffer and epoch-wise updates.

Evidence

• Setup: five benchmarks × four backbone LLMs, under an equal-budget protocol against the SkillOpt naive gate.
• Regression rate: lower in 19 of 20 settings and equal in the remaining one. With DeepSeek-V4, LiveMath falls from 36.5% to 0% and OfficeQA from 42.8% to 0%.
• Final score: highest in all 20 settings. LiveMath rises from 34.15 to 48.78; the fetched text does not name the backbone for this figure.
• An ablation reportedly verifies that disabling both components recovers the baseline. The fetched text does not include its numbers.

How to use it

The gate needs only a binary per-item verifier and paired runs of the incumbent and candidate on a fixed validation set. You can implement it from the equations above.

• The fetched text gives no code, repository or package.
• It gives no chosen values for λ, τ or α. It says only that they are tuned on held-out data, with the test set unseen throughout optimization.

Limits

• Truncated page: the fetched text stops at the accept rule (Eq. 11). It omits the per-setting results tables, the ablation, the hyperparameter values and any authors' limitations section.
• Names missing: only four benchmarks (LiveMath, OfficeQA, ALFWorld, SearchQA) and one backbone (DeepSeek-V4) are named.
• Execution noise: pairing controls shared item difficulty, but stochastic execution noise can remain, as the authors acknowledge.
• Cost: each candidate requires evaluating both skills on every validation item. The text does not say whether the equal-budget protocol charges for incumbent re-runs.
• Small validation sets: the gate is conservative by design, so with few discordant items it will abstain often. This may slow progress, and the fetched text does not quantify that cost.