Image Classifiers are Efficient Self-Supervised Video Representation Learners

What it claims

VideoMSN adapts Masked Siamese Networks (MSN) to video by tiling sampled frames into a 2D "super image" (the SIFAR layout) and training an image ViT on it, with no decoder. The efficiency figures come with named baselines: on Kinetics-400, the DINO-v3 variant matches or beats prior methods with 160× fewer video pretraining epochs than VideoMAE and 60× fewer than SMILE. The fetched text does not say which baseline the abstract's "32×" refers to. The authors call the work an integration of super images, image-pretrained ViTs and MSN, "rather than a fundamentally new architectural paradigm."

How it works

The key enabler is that the design has no decoder. An off-the-shelf image-pretrained encoder (DINO-v3 or DeiT-v3) can therefore be used as-is. Reconstruction methods such as VideoMAE usually train from scratch, partly because no image-pretrained decoders exist.

1. Super image. Sample M equidistant frames from non-overlapping segments and arrange them row-major into a grid. A square grid works best.
2. Target view. The super image, patchified and augmented but unmasked. It goes through an EMA-updated target encoder.
3. Anchor views. These go through the trained encoder: - Random view: spatial tube masking. A patch location is masked in every frame or in none, which avoids leakage across frames. - Fast view: about M/2 whole frames are dropped. Slow view: a further half is dropped, leaving about M/4 frames. The counts are approximate so the grid stays square. - Focal masking is then applied to the fast and slow views. A contiguous region is kept in every frame and everything around it is masked. This yields n fast-focal and n slow-focal views.
4. Loss. The embeddings of both views are softly assigned to learnable prototypes. A cross-entropy loss aligns each anchor's distribution with the target's, plus an entropy-maximization term. Masked tokens are dropped rather than encoded, which is where the compute saving comes from.

flowchart LR
  S[Super image] --> T[Target view, unmasked]
  S --> R[Random tube-masked view]
  S --> F[Fast view ~M/2 frames]
  S --> W[Slow view ~M/4 frames]
  F --> FF[Focal masking]
  W --> WF[Focal masking]
  T --> E[EMA target encoder]
  R --> Enc[Shared ViT encoder]
  FF --> Enc
  WF --> Enc
  E --> P[Prototype assignment]
  Enc --> P
  P --> L[Cross-entropy + entropy reg.]

The paper claims that whole-frame temporal dropping, on top of spatial masking, learns better representations than spatial masking alone. The supporting ablation is not in the fetched text.

Evidence

The fetched text gives only the Figure 1 comparison on Kinetics-400 top-1, plotted against pretraining epochs. Baselines include VideoMAE (800/1600 ep), SMILE (600 ep), MGM (800/1600 ep), MME, MGMAE, CMAE-V, OmniMAE, ST-MAE, ViC-MAE, SVT, BEVT and SIGMA-DINO. Exact accuracies cannot be read from the extracted text. The paper says experiments cover four benchmarks but names only Kinetics-400, UCF101 and HMDB51.

Limits

• The fetched page cuts off partway through Section 3.3. No result tables, per-dataset numbers, ablations, low-shot setup, hardware or wall-clock costs appear.
• The epoch savings count video pretraining only. The image pretraining of DINO-v3 and DeiT-v3 is not included.
• No code, weights or licence are stated. Only a project page is given: https://cvir.github.io/projects/videomsn. Check there before planning to adopt it.