<aside> 🛠

Language-Prompted Video Segmentation + Multi-Object Tracking

Segment and track objects through a video using a short text phrase like "person" or "red car". SAM 3.1's Object Multiplex design tracks up to 16 objects jointly in one pass, with ~7× faster multi-object inference vs. SAM 3 at scale and improved VOS performance on 6 of 7 benchmarks.

</aside>

What it does

SAM 3.1 (Video) extends SAM 3.1's open-vocabulary, text-prompted segmentation to video - every matching object is detected, masked, and tracked across frames with persistent IDs. Type a short concept ("person", "red car", "dog") and the model returns per-instance masks for every frame in the clip. The headline change vs. SAM 3 is Object Multiplex, a shared-memory design that lets the model track many objects jointly in a single forward pass instead of running an independent tracker per object. The result is ~7× faster inference at 128 objects on a single H100 GPU, with no accuracy loss and improved VOS (Video Object Segmentation) performance on 6 of 7 standard benchmarks. The studio node tracks up to 16 objects per pass at ~1 – 3 minutes per 200 frames on typical hardware.

Problem it solves

• Track many objects at once, cheaply – Object Multiplex tracks up to 16 objects (128 internally on bigger hardware) in a single pass, far more efficient than running one tracker per object
• Language-driven video search – Find and isolate every "person", "red car", "dog", etc., across an entire video clip with a single text prompt
• Editing & VFX rotoscoping – Get per-object masks per frame for compositing, blur-out, color grading, or style transfer without manual rotoscoping
• Surveillance & analytics – Automatically track people, vehicles, or other concepts through a scene with consistent IDs
• Sports & broadcast tagging – Track all players, balls, or referees frame-by-frame with one query
• Privacy filtering in video – Continuously blur faces, license plates, or screens across an entire clip
• Dataset generation – Bootstrap labeled video segmentation datasets from minimal text prompts

Input/Output

<aside>

• Input: Video clip + text prompt (or visual prompts)

  • Format: Standard video formats (MP4, MOV, etc.); also accepts a frame sequence

  • Prompts: Short text phrases ("person", "red car", "dog") or visual prompts (points, boxes, masks) on a reference frame

  • Object capacity: Up to 16 objects tracked jointly per pass in the studio node

    ebf8274b-d767-4c55-8bae-10e545d3fe16.mp4

</aside>

<aside>

• Options:
  • Merge to binary mask video (checkbox) - when on, all per-instance masks are flattened into a single binary mask video for the concept (useful for blur-out / privacy / single-class workflows) </aside>

<aside>

• Output: Per-frame segmentation masks with persistent instance IDs

  • Masks: Binary or confidence masks for each matching object, per frame, with consistent IDs across the clip (or a single merged binary mask video when "Merge to binary mask video" is on)
  • Tracks: Bounding boxes and object boundaries over time
  • Confidence scores: Per-mask, per-frame

  8807bea6-064a-40ba-bd7d-9744ca8a326a.mp4

  

</aside>

Performance

Technical Details