Skip to content

Back to the catalog

User Model Extraction via Belief Self-Distillation
arxiv.orgpaper

User Model Extraction via Belief Self-Distillation

This paper introduces Belief Self-Distillation (BSD), a read-write probing framework where a frozen LLM acts as its own teacher to learn a compact representation of its implicit beliefs about the current user, which can be both decoded (read) and written back into the model's activations (write) to causally test its effect. The authors show BSD recovers user beliefs more faithfully and enables stronger interventions than matched hidden-state steering, and that changing the model's inferred user intent alters refusal decisions even when the request itself is held fixed. They also find that independently trained LLMs across different model families converge on a similar geometry for representing users, with direct implications for how safety-relevant refusal behavior depends on who the model believes it is talking to.

TL;DR A self-distillation probing method shows LLMs form causally-writable internal "user models" whose inferred identity/intent directly steers refusal behavior, independent of the request.
Open original ↗
Also seen from another angle🛡️ Security →
Document

What it claims

Belief Self-Distillation (BSD) trains only two linear maps around a frozen LLM: a read map A that compresses a hidden state into a 128-dim user vector, and a write map B that injects that vector back. On Llama-3.1-8B, Qwen3-8B and OLMo-3-7B, linear probes on the 128-dim vector stay within 4% macro-F1 of probes on the raw hidden state, which is 32× larger. Steering through the vector is much stronger than steering on raw states. On Llama-3.1-8B it flips 78% of belief predictions, against 21% for raw-state steering and 13% for a random vector.

How it works

  1. Belief targets. For each conversation and each of 13 attributes, the frozen model answers multiple-choice questions about the user. The 13 attributes cover socio-demographics (continent, gender, education) and safety-relevant traits (truth-seeking intent, interaction style, AI trust). Multiple templates are used, the answer options are cyclically permuted, and the option-letter probabilities are mapped back and averaged. The result is the target distribution b_a(c). MCQ is used because open-ended answers can refuse, hedge or fall outside the attribute values.
  2. Information asymmetry. A frozen teacher copy sees only the conversation. A, applied to the teacher's last-token hidden state at layer ℓ, produces v_user. A frozen student copy sees only the attribute probe and gets v_user through h_S + B·v_user.
  3. Constraints and loss. B is constrained to orthonormal columns, so injection preserves norm. A and B are trained by minimising KL(teacher belief ‖ student answer distribution). This is about 0.01% of model parameters at r=128.
flowchart LR
  C[Conversation] --> T[Frozen teacher]
  T -->|h at layer ℓ| A[Read map A]
  A --> V[v_user, r=128]
  P[Attribute probe only] --> S[Frozen student]
  V -->|B, orthonormal| S
  S --> Q[Student answer dist.]
  M[Frozen model: conv + MCQ] --> Bt[Belief target b_a]
  Bt --> KL{KL loss: trains A, B only}
  Q --> KL

Steering uses contrastive activation addition. The steering direction is the difference between the mean v_user of the target class and the mean of the source class. It is unit-normalised and injected as h + α·B·Δv.

Evidence

  • Data. About 27k WildChat conversations plus about 31k WildJailbreak prompts, with annotations discarded. The split is 50k train / 1k validation / 7k test.
  • Steering setup. 86 attribute-value pairs × 50 test items gives 4,300 steering evaluations per model. Flip rate is measured across all MCQ layouts. α is swept over a shared grid, and the best flip rate per method is reported.
  • Llama-3.1-8B flip rate. 78% for v_user, 21% for the raw hidden state, 13% for random. The 12 non-target attributes still move, at an average 37% flip rate.
  • OLMo-3-7B / Qwen3-8B. v_user beats raw-state steering by 23% and 13% respectively. Qwen3-8B's random baseline alone reaches 27%; v_user is about 2× that.
  • Write-map ablation. The paper keeps v_user and Δv fixed and swaps the learned B for a random orthonormal projector. The learned B wins on 85/86 pairs for Llama, 62/86 for Qwen and 71/86 for OLMo.
  • Open-ended generation. Qualitative examples show injection changing a story's protagonist or a film recommendation without editing the prompt.

How to use it

The code is at https://github.com/holmov1/bsd-user-models. The page gives no install commands or licence. You need:

  • white-box access to the model's hidden states;
  • two frozen copies of the model, one as teacher and one as student;
  • a hand-defined attribute set with MCQ probe templates;
  • a corpus that varies the model's user beliefs (the paper mixes WildChat and WildJailbreak).

Limits

  • Truncated page. The fetched text ends partway through §4.1. The refusal result (§4.3) and the cross-model geometry result (§4.4) are not in it, so their numbers, setup and effect sizes can't be checked here.
  • No LatentQA comparison. LatentQA is the closest prior method, but the authors compare only against linear probing and raw-state steering.
  • Off-target leakage. Steering is not clean: about 37% average flips on non-target attributes, against 78% on the target attribute.
  • Qwen3-8B sensitivity. Qwen3-8B reacts strongly to any perturbation (27% random flips).
  • Optimistic flip rates. Flip rates are best-of-α-sweep figures.
  • Coverage. Only three models of 7–8B were tested, with a fixed 13-attribute schema.
  • What "belief" means here. Beliefs are measured via MCQ answers. They reflect what the model believes about the user, not ground truth about the user.

Techniques named here

  • Belief Self-Distillation
  • hidden-state steering
  • linear and causal probing

Related in ai-safety