Exploring the Limits of the Delta Learning Hypothesis: Single-Pair Generator Biases in Direct Preference Optimization

What it claims

Building DPO pairs from one fixed generator pair (larger model's answer as "chosen") gives a dataset where surface features predict the label. The authors (Okkam, Villanova.ai, University of Trento; arXiv v1, October 2026) quantify this. A classifier that sees only a spurious binary feature f (bullet points, \boxed{}, # markers, length) recovers the chosen/rejected order with accuracy 1/2 + 1/2·|p_f(chosen generator) − p_f(rejected generator)|, where p_f is how often each generator shows that feature. The gap is constant for a fixed pair, so it does not average out as the dataset grows. Drawing generators per example from a pool turns the gap into a random variable and pushes the accuracy toward 1/2.

How it works

• Domain-conditional bias (Corollary 1). The same formula holds inside any prompt domain, such as math or code. A small dataset-wide gap can hide large, opposite-signed gaps per domain. The policy can still exploit these, because it can infer the domain from the prompt.
• Pooling result (Proposition 2). Pooling removes the bias only if generator identity varies within each domain. It also requires that the selection criterion depends only on wanted features, such as correctness or safety.
• Residual case. If quality correlates with a spurious feature across the model population (for example, stronger models write longer answers), pooling will not remove it. The authors leave this case to future work.
• Diagnostics. They check the mechanism two ways. One is the gap between reward margins and accuracy during training. The other is the accuracy of a classifier given only style features. Fixed pairs show substantially higher reward margins and pairwise accuracy whichever model is trained, which the authors read as reward hacking.

Evidence

Setup

• Data: the olmo2-1b-prefmix DPO dataset, with all completions regenerated. Chosen and rejected are assigned strictly by parameter count within one family.
• Training: length-normalized DPO, 1,000 steps, batch size 128. Learning rate and β were swept and selected on GSM8K train, the ARC-Challenge validation set and IFBench.
• Trained models: Qwen2-1.5B-Instruct, Gemma2-2B-it, Qwen1.5-4B-Chat, Qwen1.5-7B-Chat, Aya-23-8B and Mistral-7B-Instruct-v0.2.
• Main single pair: Qwen3-4B vs Qwen3-0.6B.
• Robustness pairs: Qwen3-32B/0.6B, OLMo2-32B/7B-Instruct, OLMo2-7B/1B-Instruct and Gemma2-27B/9B-it.
• Evaluation: TruthfulQA, ARC-Challenge, IFEval, MATH-500, GSM8K, HumanEval and MBPP.

Results

• Multi-model data: every trained model improves on average and on most benchmarks.
• Qwen3-4B/0.6B pair: for half the trained models performance degrades. For the other half it stays flat or lags the multi-model data by a large margin.
• Other pairs: they vary somewhat, but on average they still trail multi-model data.

The fetched text shows these results only as figures, with no per-benchmark numbers in prose.

How to use it

When building judge-free preference data, draw generators per example from a pool rather than one fixed pair. Vary them within each prompt domain. Before training, a cheap check is to train a classifier on style features of chosen versus rejected. Accuracy well above 50% signals a shortcut that DPO can exploit.

Limits

• Size of the effect: adding generators without a judge only partially closes the gap, and only for most of the models tested.
• Assumptions: the theory assumes the two completions are sampled independently. The pooling guarantee also needs spurious features to be uncorrelated with quality across the model population.
• Scale: all trained models are 1.5B–8B parameters, matched to the OLMo2-1B-SFT prompt distribution.
• Truncated text: the fetched page ends partway through Section 3.1. The style-classifier accuracies (Section 3.3.3), the training-dynamics results (Section 3.3.1), the generator-pool scaling results and the hyperparameter appendix were not in it.