TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

What it claims

TrackEverything is a 3D point tracker that tracks all visible scene points across all frames of videos exceeding 1000 frames, within 40 GB of GPU memory — the first method the authors know of to do so. Prior all-frame dense 3D trackers exhaust GPU memory beyond ~96 frames. The core idea: because video is a 2D projection of a 3D world, represent the video as a persistent world-coordinate 3D scene whose size grows with unique geometry, not frame count. On TAPVid-3D it beats all open-source all-frame dense 3D trackers by more than 20% APD on short clips, and stays competitive with state-of-the-art sparse trackers on long sequences despite tracking far more points. (arXiv 2609.30222v1; CMU + Meta.)

How it works

Processing runs in non-overlapping sliding windows of L=16 frames. Three innovations carry the method:

• Encoding into a 3D feature cloud. Each frame is encoded by a frozen DINOv3-small backbone plus a trainable ViT-Adapter head (D=384), then unprojected into world coordinates using depth/pointmaps from sensors or an estimator (VGGT-Ω). Tokens are voxelized per-frame (mean-pooled within a voxel) to cut spatial redundancy — but not across frames, to avoid merging distinct surfaces that occupy the same region at different times.
• Endpoint refiner then trajectory refiner. A recurrent transformer predicts each point's 3D position at the window boundary, its visibility, and a static/dynamic label. Only dynamic points get dense within-window trajectories (initialized by constant-velocity interpolation, refined with within-track temporal self-attention and cross-attention to the scene cloud). Static points trivially keep their positions, so dense decoding concentrates on the moving subset.
• 3D WAFT. A 3D extension of Warp-Aligned Feature Transform replaces memory-heavy 4D correlation volumes: it projects a point's current target estimate onto the target frame's 2D feature map and bilinearly samples source and target features.

The turning point is cross-window de-duplication: at each window boundary, points arriving in the same 3D voxel are merged by mean-pooling, so representation size scales with unique physical geometry rather than video length.

flowchart TD
    A[RGB window L=16 + pointmaps] --> B[DINOv3 encode + unproject to 3D feature cloud]
    B --> C[Endpoint refiner: boundary position + static/dynamic]
    C -->|dynamic| D[Trajectory refiner: dense within-window paths]
    C -->|static| E[Retain position]
    D --> F[Voxel de-dup at boundary: merge co-located tracks]
    E --> F
    F -->|next window| B

Evidence

Stated on TAPVid-3D: >20% APD improvement over all open-source all-frame dense 3D trackers on short clips; competitive with SOTA sparse trackers on long sequences. Scaling claim: tracks all points across 1000+ frame videos within 40 GB, where existing all-frame dense trackers run out of memory beyond ~96 frames. The paper frames the redundancy it removes concretely — tracking all points in a 512×512, 200-frame video would otherwise require on the order of 10^10 predictions.

Limits

This is an abstract-plus-method excerpt (through Section 3.2.1); no ablation tables, no per-benchmark numbers beyond the headline APD margin, and no released code/weights or licence are visible in the fetched text (only a project site, trackeverything.github.io). The APD figure is given as a single margin without the baseline scores. The method depends on depth/pointmaps from sensors or an estimator (VGGT-Ω), so tracking quality is coupled to geometry-prediction quality — not characterized here. Comparisons to sparse trackers are "competitive," not stated as winning.