Minimally Invasive Steering of Language Models

What it claims

MISVO is a test-time alignment method that optimizes per-token pre-logit steering vectors against a black-box reward, with a Fisher-quadratic penalty derived from the local KL geometry rather than an isotropic norm penalty. The summary names this; what the page adds is the concrete evaluation and, importantly, the second baseline it competes against — AISP, not just Best-of-N. On four frozen models (LFM2.5-1.2B, Gemma3-4B-IT, Llama-3-8B-Instruct, Phi-4 14B) MISVO wins six of seven model–task settings, but AISP is the method it edges out or loses to, and Best-of-N trails both.

How it works

Steering vectors u_t are added to the hidden state entering the frozen LM head (W,b); logits shift by Wu_t, so no backprop through the transformer body is needed. Optimization is stochastic gradient ascent (Algorithm 1) using K>1 rollouts per step with a leave-one-out reward baseline, plus the analytic regularizer gradient Σ_t u_t. The cost matrix Σ_t = F̄_t is the token-Fisher averaged over reference-policy prefixes; crucially, MISVO estimates it once from the first K rollouts (which follow the reference policy since u=0 at init) and reuses it, so regularization adds no extra generations. Fisher–vector products are computed matrix-free at O(Vd) per product via C_p v = p⊙v − p(p⊤v). Theorem 1 decomposes the exact sequence-level KL gradient into the analytic Fisher term plus a suffix score-function term shown to be second-order in ‖u‖, justifying dropping it near the origin.

Evidence

SHP (500 prompts, scored by Skywork-Reward-V2-Qwen3-0.6B, best-of-KN reward, 3 seeds, 512-token cap). Reward (higher better):

Diversity and coherence stay near BoN throughout. MBPP+ (first 120 problems, reward = fraction of held-out unit tests passed, three smaller models) is described but its table is not in the fetched text.

Limits

• AISP beats MISVO on Llama-3-8B (11.07 vs 10.80) — the one non-winning setting — and generally posts higher diversity; MISVO's advantage is not uniform.
• All guarantees are local and for fixed horizon T: first-order KL-gradient agreement holds near u=0, with no bound on accumulated error along the optimization trajectory or for large interventions. Neither KL nor its Fisher quadratic globally bounds the other.
• Vocabulary-sized intermediates add working memory; runtime and peak memory are deferred to an appendix not in the fetched text. MBPP+ numbers likewise not shown here.
• Preprint (arXiv:2609.30218v1); no license or code pointer in the fetched text.