Geometric Calibration of Supervised Fine-Tuning

What it claims

SPACE is a post-hoc edit applied to a fully fine-tuned checkpoint. For each weight matrix, it removes the part of the SFT update that acts on the dominant input directions of the pre-SFT weight, and keeps the rest of the update. On full-parameter math SFT, it beats eight post-hoc refinement baselines on average in-domain and out-of-domain score: WiSE-FT, TIES, LiNeS, DARE, POME, FAPM, DG-Hard and MonoSoup. The gain over the best baseline is sometimes large and sometimes very small (see Evidence).

How it works

For every 2D weight matrix W₀ (pre-SFT) and its update ΔW = W_SFT − W₀:

1. Take the SVD of W₀ and keep the leading k right singular vectors V₀,ₖ. This gives the projector P₀ = V₀,ₖV₀,ₖᵀ, which spans the "principal input subspace."
2. Choose k separately for each matrix. It is the smallest k whose squared singular values reach a fraction ρ of total spectral energy. The default is ρ = 0.5.
3. Set W_cal = W₀ + ΔW(I − P₀). Inputs inside the principal subspace get the pre-SFT behaviour. Inputs outside it get the post-SFT behaviour.

By default this is applied to all 2D matrices, including the head and embeddings. Nothing is trained and no data is needed.

Evidence

The setup is full-parameter SFT from pretrained (not instruction-tuned) bases. For Qwen2.5-7B, training used 100K NuminaMath long-CoT examples.

Average change vs. the SFT checkpoint:

Per-benchmark highlights:

• Qwen2.5-7B MATH500 went from 44.84 to 49.36.
• Qwen3-14B MMLU-Pro went from 65.90 to 69.40.
• Qwen3-14B IFEval went from 48.80 to 54.08, still far below the base model's 65.35.
• SPACE is not best everywhere. On Qwen3-14B, LiveCodeBench v2 fell from 44.81 to 43.84, and MonoSoup scored higher on AIME25 (36.67 vs 35.83).

Ablations (from a figure; values read from extracted text, Qwen2.5-7B math, listed as ID / OOD):

• ρ (energy threshold):
• 0.25: +2.14 / +1.05
• 0.5: +2.85 / +2.26
• 0.75: +1.36 / +2.43
• Calibration strength α (0 to 1): gains rise roughly monotonically. Full strength gives the best result.
• Which matrices to edit:
• Attention only: −0.04 / +0.01
• MLP only: +2.55 / +0.59
• Attention + MLP: +2.74 / +1.12
• All 2D matrices: +2.85 / +2.26
• Most of the OOD benefit therefore needs broad coverage.
• Which subspace to use: the choice matters.
• Top right singular vectors of W₀ (the default): +2.85 / +2.26
• Bottom right singular vectors: +0.03 / +2.81. This protects OOD but gives up the ID gain.
• Left singular vectors: negative ID (−1.91 for top, −0.72 for bottom).
• Taking the subspace from ΔW itself instead of W₀: at most +1.14 / +0.95.

How to use it

You need only the pre-SFT and post-SFT checkpoints. The steps are one SVD per weight matrix plus a matrix product, using the defaults ρ = 0.5 and all 2D matrices.

The paper links a code repository and project website, but the fetched text does not include their URLs, install commands or licence. The page calls the method "lightweight" but gives no runtime or memory figures for the SVDs.

Limits

• The fetched text stops early in Section 5.1. Only the math table and the ablation figure are visible. The coding, medical and "SFT variant" results claimed in the abstract, and any limitations section, are not shown.
• All evidence shown is full-parameter SFT from base models. Behaviour after LoRA or other PEFT, or starting from instruction-tuned checkpoints, is not shown.
• The defaults (ρ = 0.5, all matrices) were picked because they "work well across the evaluated settings." The ρ trade-off is visible, so a different domain may need re-tuning.
• Qwen2.5-7B AIME scores are near floor (0.4–1.7), so its ID average is dominated by MATH500, OlympiadBench and AMC23.