Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements

What it claims

Alignment is treated as a vector of per-risk exponents, not one property. For each risk r, the burden of holding a fixed safety target scales as B_r(N) = a_r·N^α_r. Against a budget proportional to N, α < 1 means scaling helps, α ≈ 1 means alignment keeps pace, and α > 1 means "alignment debt" accumulates. The threshold of 1 is not universal. For a budget proportional to N^β, such as training compute measured against parameter count, the threshold is β.

The paper reports two measurements:

• Pythia classifiers: a reanalysis of public adversarial-training data finds that defense compute to get attack success under 10% grows as N^0.60 (95% interval 0.42–0.78). That is slower than pretraining compute.
• Qwen2.5 0.5B–72B pilot:
• Truthfulness has exponent −0.05 and stated dispositions 0.48. Both count as "scaling helps", but local slopes rise toward 1 at the largest sizes.
• Sycophancy (0.89, or 0.83 with two extra seeds at 72B) is undetermined.
• A planted backdoor is undetermined. It is removed within 128–256 examples when the trigger is known, but survives blind safety fine-tuning at four of five sizes.

How it works

Three ways to measure burden, read as complementary estimates whose disagreements are informative:

• O1, failures found: the number of distinct failures a fixed red-team budget finds. The exponent of the per-correction cost must be added before comparing to the threshold.
• O2, effort to target: the safety data or compute needed to push the violation rate below ε_r.
• O3, capability cost: the alignment tax, measured at a fixed safety level.

Three alignment quantities:

• Observed: what the evaluation in use reports. Training against it drives it toward 1.
• Audited: what an extra audit reveals, modeled as an audit with detection probability p and no false positives.
• Latent true: the value with every failure known; never directly observed.

Toy model. Each round, detected failures are corrected and the corrections consume capability headroom, H = 1 − B/N. Undetected failures cost no headroom but widen the gap between observed and true alignment.

flowchart LR
  S[Scale up N] --> F[New failures ∝ N^α_r]
  F -->|detected| C[Correct: burden adds up]
  F -->|undetected| G[A_obs stays 1, A_true falls]
  C --> H[Headroom shrinks]
  H -->|grow to restore| S

Proven results:

• The largest corrected exponent, not an average, sets the long-run regime.
• Above 1, holding headroom above a floor requires super-exponential growth.
• dα_eff/d ln N = Var_w(α) ≥ 0, so small-model fits underestimate large-scale exponents.
• An audit with no false positives never underestimates true alignment.

Illustration: in a world with base exponent exactly 1, headroom is exhausted because the fastest corrected risk has exponent 1.15. A fit over the sizes a player observes returns 0.91.

How to use it

• The paper provides a pre-registrable protocol (Section 5) with estimators, a precision analysis, confounds and decision criteria. This is the reusable part for running your own size sweeps.
• Before trusting any "safety improves with scale" result, check two things the framework exposes:
• whether the budget threshold should be β rather than 1;
• whether your fit window is small enough to bias the exponent downward.
• Four browser games play these laws under the toy-model worlds or the measured exponents: www.aisafety.fun.

Limits

• No frontier claim: the author explicitly makes no claim about which regime frontier assistants are in. None of the toy model's numbers are forecasts.
• Toy-model results are artifacts of its parameters:
• The "balance" world turns into debt only because reward hacking is given a positive shift.
• True alignment falls in every world only because the never-detected category is given the largest shift.
• Theorem assumptions may not hold in real systems: additive burden, positive mixtures of fixed-exponent power laws, and audits without false positives.
• Pythia study: one proxy of one risk, under a training recipe that varies with model size.
• Qwen pilot:
• One family, one method per risk, low-rank fine-tuning and two-choice evaluations.
• The truthfulness result is strongly attenuated when the target is set relative to the starting error.
• The Qwen3 Base replication (0.6B–14B) covers only two risks, and the disposition result rests on its smallest model.
• Provenance: a single-author v1 preprint, and the fetched text cuts off partway through Section 4. The detailed pilot tables and the protocol itself were not in the text reviewed here.