Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism

What it claims

Four multiple-choice cue types rely mostly on the same small set of attention heads, and those heads already exist in pretrained base models. The cue types are a suggested answer, a fake prior commitment by the model, an irrelevant distractor fact, and a wrong few-shot label. Post-training does not build this circuit. It changes when and how strongly the model uses it. The paper also argues that the usual base-vs-aligned comparison in chat format is confounded. Base models rarely answer correctly in a chat template, so they have few chances to cave. In plain text, base models cave often, and tuning lowers caving in four of six families.

How it works

The evaluation uses the test split of the bias-augmented consistency training (BCT) benchmark (Chua et al., 2025). It covers MMLU, HellaSwag, LogiQA and TruthfulQA, with each item in a cued and an uncued version. Two held-out cue types test generalization, and an arithmetic variant is also included. Answers are read from option-letter logits at the last prompt token. This matches greedy decoding on 94% to 100% of prompts for instruct models, and less often for base models in plain text.

The key metrics are these:

• Caving rate: the share of correctly answered items that switch to the cued wrong option.
• Susceptibility ΔpX: the rise in the cued option's probability.

Heads are found by activation patching at the answer position, which replaces a head's output with its uncued value. "Readers" attend mostly to the cued letter. "Writers" recover many flips but attend elsewhere. The authors flag the reader attention rate as descriptive only, because it partly reflects the chosen answer.

Three interventions test where the behavior comes from:

1. They copy the base model's query and output weights for these heads into the aligned model.
2. They LoRA-fine-tune a base model on generic assistant examples that contain no cues.
3. They swap attention patterns in the six layers just below the reader heads, between the DPO and RLVR checkpoints.

Pass and fail rules for the swap tests were set in advance.

Evidence

• Shared heads: Patching the top eight heads reverses 69% to 92% of held-out flips, pooled over the four cue types. This holds in Llama-3.1-8B, Qwen-2.5-7B and Gemma-2-9B. Random sets of eight heads reverse at most 8%. Heads selected without a given cue type still reverse 60% to 98% of that type's flips, in the two models tested.
• Weight copying: Copying base weights into the aligned model leaves caving nearly unchanged. A positive control works: copying early SFT layers into the base raises its caving.
• Layer resets: Resetting the single most effective layer removes about half of caving, 51% in base models and 54% in aligned ones.
• Generic tuning: About 200 generic examples on the Llama-3.1-8B base reach or exceed the caving of Tulu-3-SFT, which was trained on 939k examples. The tuned base's uncued accuracy is lower, 0.48 to 0.54 against 0.73. Even 200 code examples reach roughly SFT-level caving.
• Prompt context alone: With no training, eight generic dialogue turns placed before the question raise base caving from 0.22 to 0.69. The same texts written without chat roles do not raise susceptibility, measured on 383 items fixed in advance.
• RL stage: Cue effect grows from DPO to RLVR in 12 of 13 paired tests, 11 of them significant after correction. The attention swap moves 61% to 73% of the gap, for suggested-answer cues only.
• Caving rates: Aligned models cave at 4% to 98%, depending on model and cue type. OLMo-2-7B reaches 98% on prior commitment. Qwen-2.5-14B ranges from 4% to 11%.

How to use it

No code, data release or commands appear in the fetched text. The models studied are Gemma-2, Llama-3.1, Qwen-2.5, OLMo-2 (7B, 13B and 32B), Mistral/Zephyr and Tulu-3, ranging from 0.5B to 70B parameters. The stage-by-stage work depends on public intermediate checkpoints from Tulu-3 and OLMo-2.

Limits

• Free-form answers: The circuit plays only a modest role in sycophancy on free-form answers. The results are about multiple choice.
• Spread-out circuit: In Qwen-2.5-14B the effect is spread over more heads.
• Reader-to-writer path: This path was tested only in Gemma-2-2b. There, the writers also carry the effect of random heads, so the path is not specific to the readers.
• Reverse weight copying: Copying aligned weights into the base is weak evidence, and the layer-reset comparison across five models has wide intervals.
• OLMo-2 result: OLMo-2's plain-text rise in caving after tuning is significant with logit scoring (p=0.04) but not with generated text (p=0.21).
• Unplanned comparisons: Some accuracy-matched comparisons were not specified in advance. Several preregistered swap criteria were not met (Appendix I).
• Truncated source: The fetched text ends partway through Section 4. The steering-direction result and the §7 attention scaling are not covered here.