VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

What it claims

Table 1 shows that verified "useful" history gives coding agents only a small, statistically weak benefit, even when it is injected directly. None of the five transfer solvers has a 95% CI that excludes zero. Part of the step savings also appears when an irrelevant passage is injected. The authors trace why memory systems fail mainly to the form of the supplied record. A strip ablation ties this to transcript volume rather than to what the instructions say. They also find that a record's value depends on solver headroom: the same record can help a weaker solver and hurt a stronger one.

How it works

SIEVE construction runs offline:

1. Source validation. Drop SWE-rebench V2 instances whose gold patch fails the declared tests.
2. Patch-pattern screen. Only earlier same-repo instances are eligible, and target identifiers are excluded. A deterministic score over shared files, directories and cleaned identifiers shortlists up to 24 candidates, then an LLM judges repair-pattern similarity. The target gold patch is used only here. The test patch stays evaluator-only.
3. History runs and distillation. Each screened history task is run with a fixed MiniSWEAgent configuration. One experience is distilled per pair: bug class, root cause, fix pattern and reusable lesson. Distillation never sees the target gold patch, test patch or test outcome.
4. Retention by uplift. deepseek-v4-flash with MiniSWEAgent runs 4 seeds per condition. A target is kept only if its memory-off resolve share is below 1 and the best shortlist combination improves on it. That argmax combination is frozen.

Evaluation then has two layers on the same fixed agent configuration:

• Frozen verified experience transfer. The frozen experience is given to held-out solvers.
• Memory-system retrieval. Mem0, SimpleMem, MemoryOS and A-MEM each ingest the target's full eligible history, using deepseek-v4-flash to extract records and Qwen3-Embedding-4B for embeddings. A declared target query retrieves the top-1 experience, which is injected before the solver starts.

Evidence

Table 1 reports Resolved over 111 targets × 4 seeds, memory off → frozen experience, with the 95% CI of the difference:

The irrelevant-memory control injects a fixed passage of everyday sentences with no software content. It was run on deepseek-v4-pro, glm-5 and kimi-k2.7-code:

• It lowered Resolved on all three: 63.7% (−3.4), 54.3% (−0.7) and 69.1% (−0.7).
• It still cut steps on deepseek-v4-pro (−3.2) and glm-5 (−7.8). Those cuts match or exceed the frozen-experience cut on deepseek-v4-pro (−2.7).

So step reductions are not clean evidence that the content helped.

Token effects are mixed. Input tokens fall on glm-5 (2.45M → 1.93M), but qwen3.8-max output rises by 5.1k per run.

Benchmark profile:

• Language: Python 50.5%, JS 14.4%, Go 10.8%, Rust 9.0%, other 15.3%. Aggregates mostly reflect Python.
• Task type: Bug fixes are 29.7% of targets, with the rest spread across feature, API, parser, error handling and config work.
• History per target: This varies widely. 45 targets have 10 or fewer earlier trajectories, and 2 have more than 200.

How to use it

Code: https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/VibeMemBench

Each released target includes provenance, its test command, its environment and its execution result. The manifest records the retrieval configuration and the query template. The page gives no install commands and no licence.

Limits

• Selection bias. Targets were chosen because one reference solver (deepseek-v4-flash) benefited, and the seed-level argmax inflates that reference uplift. The transfer results inherit this selected population.
• Weak certification. The argmax covers only the shortlist, so it shows that a useful experience exists, not that it is the best one.
• Offline retrieval only. Retrieval is single-shot and happens before the solver starts. It is not closed-loop memory.
• Narrow cost metrics. Tokens and steps measure solver usage only. Latency and memory-system cost are not measured.
• Partial control coverage. The irrelevant-memory control and the memory-system layer cover only three of the five solvers.
• Truncated source text. The fetched text ends at Table 1. Per-system memory-retrieval numbers and the strip-ablation details are not available here; check the full PDF.