From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory

What it claims

ResMem trains a parametric memory to output an additive logit residual on top of a frozen LLM. The residual is fit so that base plus residual matches the same LLM reading BM25-retrieved evidence. Prior memories such as MLP Memory and Memory Decoder learn to imitate a kNN retriever on their own and meet the base model only at inference. The authors (MBZUAI, McGill, Mila, Salesforce) argue that this lets a memory that fits the retriever well still override correct base answers. ResMem's loss is computed on the joint prediction, so positions where the base already matches the target give almost no gradient. Code and weights for a Mistral-7B-v0.3 version are released.

How it works

The memory uses the MLP Memory architecture. It is a stack of gated feed-forward blocks with RMS norm, a final norm and its own trainable vocabulary projection. It reads the hidden state at the input to the final Transformer block's feed-forward sublayer. It has no attention, because that hidden state already carries the context. Only memory parameters are trained.

Training runs in two phases:

1. Offline teacher construction. For each corpus span, retrieve passages with BM25. Run the frozen LLM on [evidence; span] under teacher forcing. Cache the top-64 token probabilities per position, renormalized, with zero mass elsewhere.
2. Residual self-distillation. Run the frozen LLM on the span alone. Add the memory's logits to the base logits, then apply softmax. The loss is α·KL(teacher ‖ joint) plus (1−α)·cross-entropy on the next token. Forward KL is required because the top-K target has zero mass outside K tokens.

At the optimum, the residual equals the log-ratio between target and base prediction. At inference, the final logits are base logits plus γ times the residual. Here γ=0 recovers the base, and γ=1 gives the fitted target, with geometric interpolation in between.

The paper's override analysis explains the drop in harms. Under the additive readout, flipping a confident base token needs an equally confident target in log-odds. Under MLP Memory's linear probability mixture, a memory that puts all its mass on one token overrides any base margin once λ > 1/2. Keeping confident base answers therefore forces a small λ.

Evidence

• Storage. On an English Wikipedia subset, retained construction storage drops from 602.0 GiB (kNN datastore plus FAISS IVF-PQ index plus targets) to 32.0 GiB (text plus BM25 index plus targets). This is an 18.8× reduction. The formula gives 8,652 bytes per position for kNN with d=4096, 64-byte PQ codes and K=64, against 384 + b bytes for ResMem, where b is the text and BM25 index per position. The ratio does not depend on corpus size and has a ceiling of 22.5×.
• QA. Across five knowledge-QA datasets, ResMem improves EM and token F1 over the frozen base on every dataset. Mean F1 rises from 32.83 to 34.48.
• Against MLP Memory. ResMem corrects 11% more base errors and breaks 21% fewer correct base answers. The authors attribute most of the gain to avoided harms.
• Shared-teacher ablation. With the teacher held fixed, the residual objective mainly preserves correct answers. The extra repairs come from the evidence-reading teacher. Memory Decoder scores below both memories under the same teacher.
• Teacher shape. The teacher targets keep the shape of the base prediction, while kNN targets are much sharper.

How to use it

The base model is Mistral-7B-v0.3. The memory has 8 layers, hidden size 4096 and intermediate size 14336, for about 1.54B extra parameters. Inference needs only the base model and the memory, with no index or retrieval. γ must be tuned for the readout.

• Code: https://github.com/SSRand/From-Lookups-to-Lessons
• Weights: https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3

The fetched text does not state a licence.

Limits

The fetched text stops at the experimental setup. It does not name the five datasets, the per-dataset numbers, the baseline list beyond BM25-RAG, or the inference-cost results in Appendix D. The F1 gain over the frozen base is 1.65 points, and this text gives no comparison against BM25-RAG at inference. Training uses fixed teacher-forced continuations, not on-policy student trajectories. The memory reads the base model's hidden states, so the released weights are tied to Mistral-7B-v0.3. The paper is a v1 preprint dated October 2026.