VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

What it claims

VideoLVR trains a video MLLM to emit blocks of latent visual evidence inside its chain of thought. Each block is supervised to reproduce the encoder's own visual tokens for the regions that support one reasoning claim, even when those regions span several moments of the video. The authors (BUPT, Tsinghua, NTU; arXiv v1, October 2026) argue this does two things. It keeps visual content available during reasoning, which textual timestamps and boxes do not. It also avoids the context growth of tool calls that re-encode crops or frames.

How it works

Training data is synthesized from ordinary video QA pairs by a teacher MLLM, which the page does not name.

flowchart TD
  A[Video QA pair] --> B[Teacher writes trajectory with evidence groups]
  B --> C{Answer correct and evidence valid?}
  C -- no, errors as feedback --> B
  C -- yes --> D[Teacher relocates evidence before the claim using it]
  D --> E{Answer kept, every group once, correct order?}
  E -- no --> X[Discard]
  E -- yes --> F[Training trajectory]

An evidence group pairs a claim with one or more regions. Each region is a timestamp plus a normalized bounding box. The rejection step drops trajectories with malformed timestamps or invalid or overly broad boxes. Polishing exists because teachers tend to cluster evidence at the end of the response, which would misalign latents with the steps that use them.

To build targets, each region's timestamp maps to a temporal position in the encoded video. Its box is projected onto the merged token grid, and the intersected cells are taken in raster order. Regions in a group are sorted by time and concatenated without separators. The result fills L_k latent positions between <bot> and <eot> tokens. The textual coordinates are removed from the generation target.

Visual token prediction (VTP) replaces plain MSE regression. A two-layer MLP g_φ maps each hidden state to a query. Each of the N visual tokens of the input video is scored by negative squared distance, scaled by feature dimension d and temperature τ. Cross-entropy is then taken against the target index. The stated rationale is that adjacent frames yield near-identical tokens, and MSE does not push a prediction away from those look-alikes. The total loss is L_text + λ·L_VTP. At inference the same g_φ projects each hidden state into a latent that is fed back as the next input. No frames or crops are inserted.

For RL, TVPO samples visual tokens from the VTP distribution over the video's vocabulary. It treats them as actions alongside text tokens and optimizes both with final-answer rewards.

Limits

The fetched text ends partway through the VTP loss definition. It contains no benchmark names, scores, baselines, ablation results, base model, model size, training compute, or TVPO details, so the "fine-grained gains" claim cannot be checked from this page. No code, weights or licence are mentioned. The method needs grounded trajectories from a capable teacher MLLM, and a latent block can only point at tokens already present in the encoded input. Evidence missed by frame sampling or token merging therefore cannot be recovered. The related-work section cites diagnostic studies (Viveiros et al. 2026; Zhang et al. 2026b) showing that latent-reasoning models may bypass latents or drift from their targets. The visible text does not show whether VideoLVR was tested for this.