As the Story Unfolds: Watching a Film and Identifying Characters as a Human Does

What it claims

The Movie Character Naming (MCN) dataset has 23,792 multiple-choice naming questions over 407 films, built on MovieNet and Movie101. It also has first-appearance and first-naming timestamps for 287 characters across 20 films. The authors (Oxford VGG and Shanghai Jiao Tong University) fine-tune an open-source MLLM for naming with an SFT then GRPO recipe. They compare it against proprietary MLLMs such as Gemini. They then use it in a shot-by-shot streaming pipeline that builds a character bank of face, name and voice triplets. The paper is a NeurIPS 2026 paper, posted to arXiv on 27 October 2026.

How it works

Preprocessing builds character-level face-track clusters and speaker-diarized voice clusters. TalkNet synchronization scores assign each voice cluster to the face cluster it matches best. Parakeet transcribes the speech, which yields subtitles tagged by speaker. Clips whose subtitles contain a character name become questions. The answer is one face track, or "none" when the named person is off-screen.

The model sees colored bounding boxes drawn on the raw frames, following Xie et al. Each subtitle line carries the speaker's color and timestamps, for example <693.2s> [BLUE]"..."<695.0s>. The model is asked "Which color corresponds to X?"

Training has two stages:

1. SFT cold start on reasoning traces. A teacher MLLM writes these after being given the ground-truth mapping.
2. GRPO with reward R = format reward plus the count of correctly assigned names.

Naive GRPO learns a shortcut: it maps the protagonist to "red", the most frequent color. Randomly permuting colors per sample during RL removes this shortcut. The authors say this accounts for "a significant share" of the gain over the cold start.

flowchart LR
  F[Face tracks] --> L[TalkNet linking]
  V[Voice clusters] --> L
  L --> S[Parakeet subtitles]
  S --> M[Fine-tuned MLLM per shot]
  B[Character bank t-1] --> U[Bank update]
  M --> U
  U --> B

The bank update is autoregressive: B_t = φ_B(B_{t−1}, ψ(V_t)). Unnamed identities are kept, so a character can be named after their first appearance. Face tracks are the primary signal for merging new tracks into existing entries. The paper also describes an error-correction mechanism that uses repeated name mentions later in the film.

Evidence

The CN train split covers 377 films (108.13 h), 1,853 characters and 22,210 questions, of which 21,150 have a non-none answer. The CN test split covers 30 films (6.51 h), 226 characters and 1,582 hand-verified questions, of which 1,399 are non-none. Train and test films do not overlap. By evidence type, the test questions break down as follows:

• about 70% direct address,
• 16% third-party reference,
• 12% self-introduction,
• 2% on-screen text such as name badges.

The FO test set covers 20 films and 39.98 hours. Test labels were corrected by hand, then checked again where proprietary models disagreed with them. The motivating example is Juno: the protagonist appears in the first shot but is not named until 7.2 minutes in.

How to use it

The authors say the dataset and models are available at https://www.robots.ox.ac.uk/~vgg/research/StreamCast/. The fetched text names neither the base MLLM nor its size, nor a licence. Face tracks and voice-linked tracks are released for whole films as well as for clips.

Limits

The fetched text stops partway through Section 4.1. It contains no accuracy numbers for the fine-tuned model, Gemini or the bank pipeline. It also omits the error-correction details. The datasets derive from commercial films via MovieNet and Movie101, and access terms are not stated. Ground truth comes from mapping existing body-box labels to automatically extracted face tracks. Training labels were not manually verified; only the test set was.