Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

What it is

An October 2026 arXiv paper (Groningen/Leiden; code at github.com/alistvt/context-sensitive-memorization) that re-tests prefix-based training-data extraction with a retrieved document prepended to the prefix, rather than the prefix alone. It tests three open-weight instruction-tuned models: Llama-3-8B-Instruct, Falcon-7B-Instruct and OLMo-2-1124-7B-Instruct.

How it works

• Data: 50,000 prefixes drawn uniformly at random from Pile-CC, with no filtering for known-memorized content. Prefix lengths are 32, 64 and 128 tokens, and the suffix is always 32 tokens.
• Scoring: following Hayes et al. (2025), it uses the suffix log-probability ℓ = log P(suffix | context, prefix) rather than greedy-decode matching.
• Exposure: a sample is "exposed" if ℓ > τ. The main analysis uses τ = −16, which the paper equates to a 0.5 per-token log-prob drop over 32 suffix tokens. Threshold-sensitivity results are in its appendix A.
• Contexts (paired per item):
• empty;
• relevant: the nearest E5-retrieved document;
• irrelevant: the farthest E5-retrieved document.
• Retrieval pool: 67,655 documents from the Pile's StackExchange, PubMed and Wikipedia subsets.

Findings (from the portion of the text available)

• Longer prefixes raise suffix likelihood, and the gap between empty and contextualized scoring shrinks as prefix length grows.
• Jaccard overlap of exposed sets (empty vs relevant) rises with prefix length. Per-model values are listed in the figure in the order Llama, OLMo, Falcon; the figure's row labels are garbled, so this ordering is an assumption.

• Empty vs irrelevant gives similar values, from 0.64–0.73 at prefix 32 to 0.91–0.95 at prefix 128. Irrelevant vs relevant overlap is highest, up to 0.97.
• The paper states that context-induced flips concentrate near the threshold. Some samples are suppressed and others become newly exposed. Prefix-only evaluation misses the newly exposed ones.

What to do

• When auditing a RAG or instruction-prompted deployment for memorization leakage, don't rely on isolated-prefix extraction numbers alone. Score the same items with realistic retrieved or system context as well, and compare exposed sets item by item.
• Don't treat a lower aggregate extraction rate under RAG as evidence of safety. The robust core remains, especially with long prefixes.
• Check that your conclusions hold across thresholds, since the boundary samples are the threshold-sensitive ones.

Caveats

• Only one retrieved document is used per item (nearest or farthest), and scoring is log-probability based, not actual generation.
• Only three ~7–8B models are tested, and the data is web text (Pile-CC). Results may not transfer to larger models or other data.
• The fetched text is truncated before the sections on the core/boundary analysis (§5.2) and on generic versus semantic context effects. The numbers for those are not covered here.