MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

What it claims

MemPilot keeps two views of an agent's history. One is a query-agnostic memory bank that any off-the-shelf memory system can build. The other is the raw multimodal history, kept for curation at query time. A text-only orchestrator LLM, trained with GRPO under a preference vector λ = (λ_Q, λ_C, λ_L), decides step by step how to spend runtime compute across quality, monetary cost and latency. λ_Q is fixed at 1 throughout. The fetched text stops before the results section, so no benchmark numbers are available here.

How it works

• Dual-view memory. The bank M is replaceable, so memory construction stays separate from orchestration. The raw history H is split into fixed-length token chunks, and each chunk keeps its original multimodal context.
• Action space. At each step the policy emits either Retrieve(r, k) or Curate(r, k, i, m, v). Several actions can run in parallel within one step. A run lasts at most T_max steps.
• Retrieve does semantic-similarity search over M.
• Curate retrieves raw chunks and sends them to a selected model m from a pool of LLMs and VLMs.
• Curate arguments. The retrieval query r ("what") is kept separate from the curation instruction i ("how"). For example, r is "Lena's dress options" and i is "extract the two dress types she considered". The evidence count k replaces a fixed top-k. The flag v decides whether raw images go to a vision-capable model.
• Model routing. The orchestrator never sees pixels. For routing it gets anonymized model IDs plus short profiles giving parameter scale and supported modalities.
• Rewards.
• Quality Q is a task metric such as F1.
• Cost C is the token cost of the orchestrator plus the token cost of delegated calls.
• Latency L is a proxy estimator, not wall-clock time, because wall-clock is too noisy for stable training. Details are in Appendix A.1.
• Advantage.
• Each objective's advantage is normalized separately, then combined: A^λ = λ_Q·A^Q + λ_C·A^C + λ_L·A^L.
• Per-step credit adds η·Δu. Δu comes from answer "probes" run on the evidence prefix before and after step t, and their difference estimates that step's marginal utility.

flowchart TD
  Q[User query + evidence so far] --> P{Orchestrator policy}
  P -->|Retrieve r,k| M[Memory bank M]
  P -->|Curate r,k,i,m,v| H[Raw history H]
  H --> S{Select model m}
  S --> L[Text LLM]
  S -->|v: include images| V[VLM]
  M --> O[Append observation]
  L --> O
  V --> O
  O -->|until stop or T_max| P
  P -->|stop| A[Final answer]

Evidence

The page says that experiments on five multimodal agent-memory benchmarks give broader performance–cost–latency frontiers than trade-off-aware baselines. It also reports behavioral shifts in model choice, visual access and compute allocation as λ changes. The truncated text names neither the benchmarks nor any scores.

How to use it

The page links a project website, a GitHub repository and a "MemPilot Collection", but the extracted text does not give their URLs. It gives no install commands, orchestrator model size, licence or training compute.

Limits

• No results here. The text ends partway through Section 3.3, so this document cannot report any number, baseline or benchmark name.
• Latency is modelled, not measured. Reported latency gains may not match your serving stack.
• Cost depends on the model pool. Cost and latency trade-offs are tied to the pool's pricing and profiles, which the page shows only as anonymized IDs.
• Fixed quality weight. λ_Q is always 1, so preferences only move the cost and latency weights.
• Raw history must be kept. Curate works over H, so you need to retain the full multimodal history, images included, alongside your memory bank.