EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding

What it claims

EC-RAG is a training-free retrieval layer for open-source video-language models. It builds a chain of events from a long video and retrieves at the event level instead of the frame level. Applied to five open-source LVLMs, it raises Video-MME accuracy by an average of 4% over the same models without it. The authors also report that it consistently beats the frame-level retrievers Video-RAG and TV-RAG, but the fetched text gives no numbers for that comparison. It uses no proprietary APIs, which the authors contrast with agent-style systems such as VideoAgent and MM-VID. They describe those systems as high-latency and API-dependent.

How it works

flowchart TD
  Q[User query] --> D[LVLM query decoupling]
  D -->|R_asr or NULL| A[Whisper ASR → Contriever → FAISS]
  D -->|R_ocr or NULL| O[EasyOCR, confidence filter → Contriever → FAISS]
  D -->|R_det or NULL| K[CLIP × variance keyframe selection]
  K --> P[APE detection → scene-graph text]
  A --> E[Event chain over 30s units]
  O --> E
  P --> E
  E --> L[Query-guided event localization]
  L --> M[LVLM: chain summary + located events + raw evidence]

1. Query decoupling. The backbone LVLM reads the query, guided by a few-shot prompt, and does not look at any frames at this step. It emits up to three retrieval requests: phrases to find in the speech transcript, keywords to find in on-screen text, and objects to detect. Any of the three can be NULL. Detection requests are restricted to physical entities, so the detector is never asked to find something like an emotion.
2. CLIP-Variance keyframe selection. Each frame gets a score equal to its CLIP similarity to the detection request multiplied by its pixel-intensity variance, normalised by the maximum variance in the video. Frames whose score passes a threshold τ_s are kept, subject to minimum and maximum frame counts. The variance term is meant to reject black screens and static backgrounds.
3. Extraction. - Speech: Whisper transcribes the audio into timestamped segments, which are embedded with Contriever and indexed in FAISS. - On-screen text: EasyOCR reads each sampled frame, results below a confidence threshold τ_ocr are dropped, and the rest go into a separate FAISS index. - Objects: APE, an open-vocabulary detector, runs on the selected keyframes. Its boxes are converted into natural-language location, count and pairwise-relation descriptions.
4. Event chain. The video is cut into 30-second units, which are grouped into semantically coherent events. Each event carries its multimodal evidence and explicit temporal relations to other events.
5. Answering. The system localizes the events relevant to the query. It then passes the LVLM a summary of the whole chain, the located events and the raw evidence.

Evidence

• Stated result: an average gain of 4% on Video-MME across five open-source LVLMs, measured against each LVLM with no augmentation.
• Not in the fetched text: which five LVLMs were used, per-model numbers, and any MLVU or LongVideoBench figures.
• Motivation: the authors cite Video-MME results showing that accuracy falls once a model is given more frames than some threshold. They use this to argue that retrieval works better than simply extending the context window.

Limits

• Truncated page. The fetched text stops partway through the event-chain construction section. As a result it does not show:
• how 30-second units are merged into events;
• how inter-event relations are computed;
• how event localization scores events;
• any results tables or ablations;
• any code or licence information.
• Unset parameters. The thresholds τ_s and τ_ocr and the minimum and maximum frame counts are not given.
• NULL detection requests. Keyframe scoring depends on the detection request. The text does not say how frames are selected when that request is NULL.
• Pipeline weight. Each query requires an extra LVLM call for decoupling. It also runs four further models: Whisper, EasyOCR, APE and CLIP, plus Contriever for embeddings. The fetched text reports no latency or compute cost for any of this.
• Variance heuristic. Variance is computed on raw pixel intensity. That favours visually busy frames, whether or not the detail in them is relevant to the question.