Skip to content

Back to the catalog

github.comrepo

Show HN: Open-source, real-time object tracking for live video feeds

tamserve is an Apache-2.0 Python serving engine for SAM 2.1 and EdgeTAM video object tracking. One model copy on a GPU serves many live cameras through continuous batching, as vLLM does for chat. It reports 2.2 GB GPU memory for 8 cameras against 8.6 GB for stock code, with DAVIS 2017 accuracy of 0.8685. Tracker and PeopleTracker add name-based tracking via SAM 3, and a Router spreads cameras across GPUs.

TL;DR Serves SAM 2.1, EdgeTAM and SAM 3 video tracking for many live cameras from one shared GPU model using continuous batching.
Document

What it claims

tamserve runs Meta's EdgeTAM and SAM 2.1 trackers for many live cameras from one model copy per GPU. On one RTX 4090 with EdgeTAM, compared with EdgeTAM's own code (compiled, one copy per camera, sharing the GPU), it reports:

  • 219 vs 177 frames/s total at 1 camera.
  • 323 vs 280 frames/s at 8 cameras.
  • 321 frames/s at 64 cameras in 5.7 GB, where stock would need about 70 GB.
  • 0.28 GB for an 800-frame video, against 9.8 GB, and flat however long the stream runs.

With the larger SAM 2.1 models it matches the speed of Meta's current SAM 2 code at about a third of the memory. Objects can be added on any frame mid-stream, which stock code cannot do.

How it works

Each step runs the standard SAM 2 modules once over every object on every camera:

flowchart LR
    A[Newest frame per camera] --> B[Image encoder, batched]
    B --> C[Memory attention, row per object]
    M[(Per-object memory)] --> C
    C --> D[Mask decoder]
    D --> E[Masks out]
    D --> F[Memory encoder]
    F --> M

The page lists five changes, none of which alters the model's computation beyond floating-point rounding:

  • Every object with a new frame is one row in a shared batch (continuous batching).
  • Memory is capped at what the model reads: the prompt frame, the last 6 frames and a summary of the last 15.
  • CPU-GPU synchronisation points per step went from twelve to one.
  • One position encoding is shared rather than one per frame. This made steps 4 to 7% faster and nearly halved memory at 64 cameras.
  • Batches are split into power-of-two pieces, and all five sizes are compiled up front, so nothing compiles mid-stream.

Frames are resized on the GPU (about 0.5 ms vs 16 ms on CPU), which can differ from stock by 1 in a pixel value.

Evidence

End-to-end loop. A full loop from video to masks runs at 273 frames/s for eight 720p cameras (34 per camera) and 238 at 1080p. These figures use one object per camera on an RTX 4090. Each extra object adds about 2.4 ms per round.

Router. Two RTX 4090s reached 476 frames/s on 32 cameras, against 241 on one.

Tracker (name-based). Tested on 117 SA-Co/VEval videos (224 video-phrase pairs):

HOTAGPU time per frame
SAM 3's own tracker59.4101 ms
Tracker with SAM 2.1 large56.341 ms
Tracker with EdgeTAM54.626 ms

For 16 cameras, Tracker memory holds at 6.8 GB after 19 minutes.

PeopleTracker. On 116 home videos it scores HOTA 89.6 at 8.9 ms per frame, against 87.8 at 126 ms for SAM 3. On a 3,000-frame video SAM 3 ran out of memory at frame 423, while PeopleTracker held at 3.5 GB. On crowded streets PeopleTracker scores 72.6 by default, or 78.3 when it looks at every frame, against 77.7 for SAM 3.

How to use it

pip install tamserve

Requirements are Python 3.10+ and an NVIDIA GPU. The CPU works but is too slow for live video. EdgeTAM (54 MB) downloads to ~/.cache/tamserve on first load.

The API is a loop the caller drives. tamserve does not open cameras.

  • engine = Engine(build_video_predictor("cuda"))
  • engine.step(frames, prompts) takes one RGB frame per camera and returns masks[camera][object].
  • Prompts can be Box(left, top, right, bottom), Clicks([(x, y), ...]) or a boolean mask.
  • engine.drop(camera, obj) stops tracking. engine.add(...) starts an object on the frame just processed without advancing the others.
  • Model names are sam2.1-tiny, -small, -base-plus and -large.
  • exact=True plus prepare(frame) gives masks bit-identical to stock.

Compilation trades start-up time for speed:

SettingFirst startLater startsSpeed
compile=Trueabout 20 minabout 95 sfastest
compile="default"under 3 minabout 1 min3 to 8% slower at 4+ cameras, half speed at 1
off2 s2 s55 frames/s at 1 camera, about 200 at 4+

Set TORCHINDUCTOR_CACHE_DIR to a persistent folder, because the default cache is in /tmp. Compilation is tested on Linux only.

Name-based tracking needs SAM 3:

pip install "tamserve[sam3]"
hf auth login

You must accept Meta's licence on the facebook/sam3 page on Hugging Face. The weights are 3.5 GB.

Router(compile=True) takes RTSP addresses through add_camera and needs OpenCV. Each worker decodes its own streams.

Limits

Licences. tamserve, EdgeTAM and SAM 2.1 are Apache-2.0, but SAM 3 has its own non-Apache licence. DAVIS 2017, the accuracy benchmark, is research-use only.

Tracking behaviour.

  • Tracking runs forward only. A correction restarts the track from the new prompt; there are no refinement clicks.
  • The engine cannot find objects without SAM 3. An object that leaves the view comes back as an empty mask.
  • SAM 3 detection is not batched across cameras. Tracker totals about 30 frames/s on a 4090 with EdgeTAM, or 16 with SAM 2.1 large, however many cameras share it.
  • Tracker and PeopleTracker number objects per camera only. They do not run through a Router.
  • In crowds of 90 people, PeopleTracker follows 72% of visible people, or 84% with start=0.5, at about twice the number of ID swaps.

Hardware. All speed figures come from one RTX 4090. GPUs without bf16 support are far slower: on a Colab T4, one SAM 3 detection pass takes 3 seconds.

Organisations named here

  • DAVIS 2017

Technologies named here

  • vLLM
  • Python
  • EdgeTAM
  • PeopleTracker
  • Router
  • SAM 2.1
  • SAM 3
  • tamserve
  • Tracker

Techniques named here