4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

What it claims

4DMemoryBench is a video QA benchmark in which a single question can depend on more than 100 hours of ordered source video. The longest scope, a complete TV series, is 97.98 hours. Prior benchmarks top out at 1 to 2 hours per input: Video-MME at 1 h, HourVideo at 2 h, LVBench at 68.4 min. The corpus has 1,025 standalone web videos and 402 episodes from 2 television series, totaling 887.48 hours. The 4DMemory Harness is a model-agnostic wrapper. It builds memory once, independent of any question, and retrieves from it per question, so the task's duration no longer depends on the model's native context window.

How it works

Each item is a question, an answer format and an ordered set of source videos. The reference answer, the annotated evidence, an optional executable program and task metadata stay private. Every item has exactly one task family: visual, textual or spatiotemporal. The label is used for reporting only and is not given to agents as a routing hint. "4D" here means evolving image-plane evidence, not metric 3D reconstruction.

The QA set is built along two paths:

• Standalone videos (4,388 QA). An LLM generator receives time-ordered evidence cards (clip summaries, ASR, temporal bounds). It composes questions that link 2 to 4 clips through comparison, ordering, state change, cross-modal binding or distributed counting. A separate verifier removes each supporting clip in turn to check that every clip is needed. Each video keeps 3 to 5 accepted questions.
• TV series (826 QA). Deterministic programs compute answers over validated dialogue, ASR and visual statistics at episode, season or series scope. An LLM rewrites each program as a question, and a second call checks that the wording preserves the meaning.

The harness runs in two phases:

flowchart LR
  V[Video + ASR] --> O[Observe and store]
  O --> M[(Persistent memory, timestamped)]
  Q[Question] --> F[Foundation model]
  M --> R[Retrieve]
  F --> R --> RE[Reason] --> RF[Refine]
  RF -->|need more| R
  S[Skill library and tools] -.optional.-> RE
  RF --> A[Answer]

Memory holds events, entities and spatial relations, with timestamps mapped back to the original source timeline. The skill library covers video processing, visual grounding, text statistics and TV scopes. Memory is queried through calls such as video_memory_query.

Evidence

The fetched text gives no per-model scores. Figure 2 compares the same models' Video-MME and LVBench scores with their 4DMemory scores, and every model shown scores lower on 4DMemory. The abstract puts the drop at more than 10%. Table 1 marks only EgoMonth (4 to 5 hours per participant, calendar span) and MemLens (32K to 256K tokens, image-text) as also supporting multi-history scope and a harness.

Limits

The fetched text stops partway through Section 2.2. Results tables, the list of the 13 models, ablations, and any code, data release or licence are not shown. The stated totals disagree slightly: 887.48 h, "850+ hours" and "about 900 hours". The QA pairs are generated and verified by LLMs, so "manually reviewed" covers only the final filtering. Visual counts on TV series are tracker identities per clip summed across clips, not deduplicated people or objects, which matters when a counting answer looks wrong. The Table 1 harness column records only harness support the papers document, not whether a third-party agent could be integrated. Two TV series form the entire serialized split, so series-scale results rest on a narrow content base.