Skip to content

Back to the catalog

User Model Extraction via Belief Self-Distillation
arxiv.orgpaper

User Model Extraction via Belief Self-Distillation

This arXiv paper introduces Belief Self-Distillation (BSD), a technique where a frozen LLM teaches itself a compact, decodable representation of its inferred beliefs about the user, which can then be read out and rewritten back into the model's internal state. The authors show refusal behavior depends not just on the literal request but on the model's inferred user intent, and that directly editing this belief representation changes whether the model refuses a fixed request — a causal steering result with clear implications for jailbreak mechanisms and safety-conditioning defenses. They also find that independently trained LLMs converge on a similar internal geometry for representing users.

TL;DR New interpretability technique (BSD) shows LLM refusal is causally steerable by rewriting the model's inferred beliefs about who the user is.
Open original ↗
Also seen from another angle🧠 AI →
Document

What it is

BSD (Belief Self-Distillation), from a TUM/Helmholtz group (Kholmovaia et al., arXiv:2609.31603v1, Sep 2026). Code at github.com/holmov1/bsd-user-models. The summary has the concept; what it omits is the concrete method and the numbers that tell you whether the steering effect is strong enough to act on.

How it works

Two frozen copies of the same model. A teacher sees the conversation; a student sees only an attribute probe, never the conversation. A learned rank-r read map A compresses the teacher's hidden state into v_user (r=128, ~32× compression, ≈0.01% of params); a learned write map B (constrained orthonormal, so injection is norm-preserving) writes it into the student. Only A and B train, minimizing KL between teacher and student answer distributions. Belief supervision comes from multiple-choice probes (not open-ended generation, which can refuse/hedge), with template variation and cyclic option permutation to counter MCQ ordering bias. Steering uses contrastive activation addition: ∆v = µ_target − µ_source, injected as h + αB∆v.

Key numbers

Evaluated on Llama-3.1-8B, Qwen3-8B, OLMo-3-7B. 13 attributes across socio-demographic axes plus safety-relevant ones (truth-seeking intent, interaction style, AI trust). Trained on ≈58k conversations: ≈27k WildChat + ≈31k WildJailbreak (annotations discarded), split 50k/1k/7k.

  • Read fidelity: macro-F1 gap between raw hidden state and compressed v_user stays within 4%.
  • Write / flip rate: Llama-3.1-8B — v_user flips 78% of predictions vs 21% for raw-state steering, random baseline 13%. OLMo-3-7B +23% over raw, Qwen3-8B +13% over raw. All interventions norm-matched, so the gap is direction, not magnitude.
  • Selectivity: target attribute flips 78% while cross-attribute leakage across the other 12 averages 37% — better than raw but not clean.
  • Learned map matters: replacing B with a random orthonormal projector loses on 85/86 attribute pairs for Llama (62/86 Qwen, 71/86 OLMo).

Caveats

Qwen3-8B is highly perturbation-sensitive: its random-vector control alone flips 27%, so v_user's advantage there is ~2× random rather than a clean win. Cross-attribute leakage (37% avg) means interventions are directional, not surgical. No direct empirical comparison to LatentQA — authors cite the cost of retraining its decoder per model and treat it as complementary rather than a baseline. Preprint, under review; results are on 7–8B open models only.

Organisations named here

  • Belief Self-Distillation

Technologies named here

  • LLM

Related in AI Security