MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair

What it is

MemSecBench is a benchmark paper (arXiv:2607.27080) that treats agent memory poisoning as a lifecycle, tracing one malicious instruction through a Write → Execute → Forget protocol rather than testing whether a model merely recognizes bad content. Beyond the headline numbers in the summary, the page names the actual stack under test and the success definition, which is what an engineer needs to know whether the finding applies to their deployment.

How it works

Each of the 310 cases is a linked task package T = (ρ, x_W, x_E, x_F, ℓ): source material for a clean memory state, three agent-facing stage tasks, and evaluator-only labels/rubrics/consequence contract. A configuration is Π = (H, B, L) — agent harness, memory-backend stack, served LLM. The named harnesses are OpenClaw and Hermes; the named memory backends include native file-based memory, Mem0, and A-Mem (four total). A separate judge model, DeepSeek-V4-Pro, adjudicates; cases were authored with GPT-5.5 (distinct from the evaluated LLMs).

The central mechanism the paper illustrates is temporal-context loss: a temporary instruction ("valid through June 30") gets stored as a standing rule when the memory backend drops the expiration metadata, so a later benign task recalls it and produces a real consequence (routing a payment to an obsolete account). This is why backend choice — what each system stores, retrieves, and updates — drives the outcome variance.

An attack counts as successful only when the full chain passes: W2 (persistence) ∧ E1 (recall) ∧ E2 (adoption) ∧ E3 (case-defined external consequence). A write (W1) or persistence (W2) alone is explicitly not a completed attack. Verdicts are gated on admissible evidence — agent claims, intentions, or tool calls without artifact/service-state support cannot establish Persistence, Externalization, or Repair.

Preconditions / Affected

The threat model assumes attacker-controlled content already reaches the agent through a supported Carrier and that Write hits the intended memory interface — it does not model how the attacker gains account or session access. Notably, native memory files count as in-scope attack surface when they serve as the agent's store. Execute and Forget each run from an independent copy of the same verified poisoned state M_W.

What to do

Additional metrics past the summary that sharpen the risk picture: among successfully poisoned cases, 59.6% complete the full Execute chain and 56.1% achieve selective repair; the largest backend-vs-native gap is 16.1 percentage points for end-to-end attack success (alongside the 41.3-point repair swing). Treat memory-backend selection as a security decision, not just a utility one, and verify that your backend preserves temporal/expiry metadata on stored instructions. When evaluating your own agent, test selective repair — removing malicious semantics while preserving required benign memory — since no prior benchmark jointly verified this and it varies most across stacks.

Caveats

The paper frames its cross-backend numbers as descriptive contrasts, not causal claims. The named products (OpenClaw, Hermes, and the model IDs) appear as the paper states them; confirm mapping to your own tooling before generalizing.