RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

What it claims

Generalist open video language models fall to at most 22.2% off-by-one (OBO) accuracy at high repetition counts. Specialist counters stay accurate in the same setting. The collapse is hidden by overall accuracy. It persists when the number of frames per repetition is held fixed, which rules out frame sampling as the sole cause. A linear probe on the final-layer state the language head reads recovers the count "substantially more accurately" than the model's own answer, on both real and synthetic video. The authors therefore treat readout, not perception, as the thing to fix. The evaluation covers seven open generalist VLMs and two specialist counters.

How it works

The benchmark has two parts that share the same task, count range (k = 1–20) and metrics.

RepProbe-Real has 32,233 clips cut from 1,038 RepCount source videos (exercise footage).

• Consecutive rest-pose timestamps define each repetition as a half-open interval [start, end).
• Zero-length and non-monotonic intervals are discarded.
• k consecutive repetitions are concatenated into a clip, so every clip holds only complete repetitions and ends at rest.
• Windows are taken from the start, middle and end of each source, which varies clip position while the action stays fixed.
• Splits are made at the source-video level.
• Nested k-balanced subsets are stratified by position: 2K clips (100 per count) and 10K clips (500 per count). The 10K subset is the default for controlled analyses.

RepProbe-Synthetic has 10,000 clips, 500 per k. Every clip lasts 4 s, so duration carries no count signal. Each clip shows simple geometric elements on a plain background, with exactly one attribute changing periodically. The ten families fall into three groups:

• Appearance: Flash, Scale, Color change, Shape change.
• Motion: Bounce, Pendulum, Orbit, Crossing.
• Stress: Textured flash (cluttered background) and Multi object flash (flashes take turns across several objects).

Diagnosis. For the strongest generalists, two readouts are compared from the same final-layer state:

flowchart LR
  V[Clip with exactly k reps] --> M[VLM]
  M --> S[Final-layer state]
  S --> H[Language head]
  H --> A[Generated answer]
  S --> P[Linear probe]
  P --> C[Probed count]
  A --> X{Compare to k}
  C --> X

Evaluation protocol (for reproducing or comparing)

• Prompt, verbatim: Count the number of repetitions in this video. Return only an integer .
• A shared, model-agnostic parser extracts the integer (App. A.4). Unparseable responses are scored as a prediction of 0, so refusing to answer cannot improve a score.
• Predictions are clipped to [0, 40] for the cross-model comparison.
• Metrics: MAE, normalized MAE (|k̂−k|/k, used across count ranges), exact match, and OBO (|k̂−k| ≤ 1).
• Count bands: Few (k = 1–6) and Mid (k = 7–13). The definition of the Many band is cut off in the fetched text.

Limits

• Truncated page. The fetched text ends partway through Section 3.3. It does not name the seven VLMs or the two specialists, and it gives no per-model tables or probe accuracies. The only number available is the 22.2% OBO ceiling. Check the full paper before citing specific models.
• Real-set confounds. RepProbe-Real is skewed toward small counts because fewer sources have many repetitions. Clip duration also grows with k. The authors use the balanced subsets and the fixed-duration synthetic set to control for this.
• Narrow real content. The real data is exercise footage only. The synthetic set is the evidence that the findings generalize beyond exercise.
• Decodability is not usage. The authors state that a decodable count does not show the model uses that information. The probe result shows the count is present, not that a specific fix will recover it.
• No release details. The fetched text gives no download location, licence or code release for the benchmark.