Beyond the Timeline: Augmenting Long-Video Memory with Grounded Entity Biographies

What it claims

GEB is a long-video memory that resolves physical identity before question time. It groups visually grounded observations of the same object or person across clips into a "biography," so retrieval can follow one instance across events and days. This addresses cases like two red mugs whose captions match but whose histories differ. On EgoLifeQA it reaches 72.0% against 67.6% for MAGIC-Video, the strongest published memory framework. The comparison uses the same retrieval controller, answer model and retrieval limits, so the gain comes from the memory representation, not from a stronger reasoner.

How it works

Writing the memory

1. An open-vocabulary detector plus a within-clip tracker turn detections into observations: one tracked subject in one clip.
2. A vision-language model describes each observation from three inputs: subject crops, scene frames, and timestamped narration or dialogue. It is prompted to use the text only when the text concerns that subject.
3. Observations are processed in time order. Candidate entities are retrieved by multimodal embedding similarity. An observation joins an entity only if all three conditions hold against a bounded set of that entity's recent observations: - Its similarity to every reference is at least θ_cons. - Its similarity to at least one reference is at least θ_match, where θ_match is higher than θ_cons. - There is no "separation" evidence: in enough shared frames, the two bounding boxes must not have low IoU (be shown apart).
4. If several entities qualify, the one with the highest mean similarity wins. If none qualifies, the observation starts a new entity.
5. People follow the same process. When participants are named, an extra stage consolidates identities and abstains when identifying features are shared.

The resulting graph has four node types: entity, observation, multi-scale episode, and source clip. Its edges are entity membership, episode context, visual provenance, temporal adjacency, and containment.

flowchart TD
  O[New observation] --> C[Embedding candidates]
  C --> A{Similar to all refs AND strong match to one?}
  A -- no --> N[Start new entity]
  A -- yes --> S{Shown apart in shared frames?}
  S -- yes --> N
  S -- no --> J[Append to best entity]

Reading the memory

• Observation descriptions and episode captions share one semantic and lexical index. Initial matches come from that index and from visual matches to source clips.
• Relevance spreads through membership and context edges using Personalized PageRank, and the PageRank score is combined with query-text similarity.
• Per-entity and overall caps keep a frequently seen subject from crowding out episodic context in the retrieval budget.
• For timestamped questions, only records before the query time are used.
• Each biography excerpt also lists appearances the controller has not yet inspected, which gives it concrete targets for a follow-up search.

Evidence

• EgoLifeQA accuracy: 72.0% vs 67.6% for MAGIC-Video, under matched controller, answer model and retrieval limits.
• Evidence access: the share of questions whose evidence window reaches the answering context rises from 37.6% to 58.9%.
• Ablations: indexing the extra descriptions without instance association recovers only part of the gain. Removing any one of these reduces accuracy:
• the same-instance edges
• the observation→timeline edges
• the biography text

The fetched text gives no ablation magnitudes and no per-benchmark numbers for the other three benchmarks (day- and week-long, multiple-choice and open-ended).

How to use it

The page mentions a project page but gives no code, weights, licence or commands. The excerpt does not name the detector, VLM, embedding model or answer model. Thresholds, the IoU separation settings, relation weights and budget values are deferred to Appendix A, which is not in the fetched text.

Limits

• Matching is deliberately conservative. A wrong merge creates false paths between events, so expect fragmentation (one object split across entities) rather than conflation.
• Gaps are not absence. A gap in a biography does not mean the entity was absent or inactive.
• The separation veto needs co-visibility. It only fires when two observations share frames. A zero means "no evidence of difference," not proof of sameness. Look-alike objects that never appear together rely on embeddings alone.
• Person identity abstains when identifying features are shared.
• Truncated text. The fetched page ends mid-method, so the full results, cost figures and any stated limitations section were not available for this analysis.
• Provenance. v1 preprint, from an internship at Amazon.