Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video

What it claims

The authors (MIT, Cornell, TU Wien and Liquid AI) built a tennis dataset of 85 rallies and 53,155 egocentric frames. They fine-tuned 13 vision-encoder configurations on seven egocentric prediction tasks. Image encoders and VLMs perform comparably, and the video encoder (V-JEPA 2) does worse. Adding depth input cuts ball localization error by 50 to 60%. With image encoders, ball forecasting is just as good from one conditioning frame as from longer histories. A finetuned vision-only LFM2.5-VL-450M runs at 100 Hz on modern desktop GPUs. It drives a conditional diffusion model that re-plans whole-body tennis motion as its contact predictions update.

How it works

One expert player wore a head-mounted camera (3840×2160, 29.97 fps). Five static cameras (at least 5120×2880) recorded the same session: four at eye level on the half-court corners and one raised behind the court. About 80 minutes were recorded, which yields 29.6 minutes of active rally time. Rallies are 29 free play and 56 structured drills across four patterns. They contain 557 ego shots (328 forehands, 229 backhands) and 574 opponent shots.

Labels come from the three rear allocentric cameras:

1. An off-the-shelf tennis ball tracker and a general object detector run at 1280×720.
2. Humans corrected about 30.5% of ball frames and 5.8% of player frames.
3. A DLT solver triangulates the ball freely in 3D. The player is placed on the court plane at the bottom-centre of the bounding box.

Median reprojection error is 2.1 to 4.4 px for the ball and 3.3 to 9.7 px for the player. Cameras were synced by hand from claps, with intrinsics from a ChArUco board and extrinsics from PnP on court corners.

The models tested are:

• Image encoders: DINOv3 ViT-L/16, SigLIP 2 So400M/16, Depth Anything 3-L and SAM 3, all 300 to 450M parameters.
• Video encoder: V-JEPA 2 ViT-L/16.
• VLMs: Qwen3-VL-4B, LFM2.5-VL-3B and PLM-3B, each run as vision tower alone and with the language stack.
• Small VLM: LFM2.5-VL-450M, whose vision tower has 86M parameters.

Each task head uses cross-attention from learned queries (N queries for N-step forecasts) over the token stream, then a 2-layer MLP. All heads train jointly with the encoder. Losses are L2 (weight 1.0) for continuous targets and cross-entropy (weight 0.1) for stroke type. Training uses batch 32 for 30 epochs, learning rate 1e-4 for heads and 1e-5 for backbones, and LoRA on language models. In the vision-language setting, representations are tapped at 50% of language-tower depth with a fixed task prompt.

flowchart LR
  V[Egocentric frame] --> E[LFM2.5-VL-450M vision encoder]
  E --> C[Contact position]
  E --> T[Time to contact]
  E --> A[Ball approaching or departing]
  C --> D[Conditional motion diffusion]
  T --> D
  A --> D
  D --> M[Whole-body kinematic motion]
  V -. new frames refine estimates .-> E

Evidence

Best-model errors are 0.14 m for ego player and 0.54 m for ball localization. Forecasting errors over 1.1 s are 0.34 m (player) and 0.91 m (ball). Contact-time error is 0.14 s. The authors equate the ball error to 1 to 2 frames of ball motion and the player error to 3 to 7 frames of player motion. Evaluation uses held-out whole rallies, stratified by drill type and free play.

Effects of rally phase and horizon:

• Close to ego contact, ball error falls but player error rises.
• Ball error grows with forecast horizon.

The authors read the depth result as metric depth being weakly encoded in these representations.

How to use it

The dataset will be released "upon acceptance of the paper". It will include synchronized allocentric video, 3D ball trajectories, 2D court-frame player positions, stroke labels and contact times. The project site is https://tennis-vision.csail.mit.edu. The page gives no code, weights or licence.

Limits

The data covers one expert player, one outdoor court and one roughly 80-minute session. Labels come from triangulation, not motion capture, and ball labels needed heavy manual correction. Player position is a 2D foot-point approximation. Egocentric ball position is missing in 2.6% of frames because of occlusion. The diffusion model outputs kinematic motion only; the page shows no robot deployment. The fetched text stops at the experimental setup, so per-model result tables and the diffusion model's evaluation are not visible.