SAM 2 — Segment Anything in Images and Videos by Meta

Introduction

SAM 2 extends Meta's original Segment Anything Model to handle both images and videos in a unified architecture. It introduces a streaming memory mechanism that lets the model track and segment objects across video frames in real time, while maintaining the interactive prompting interface from SAM 1.

What SAM 2 Does

• Segments any object in a single image from point, box, or mask prompts
• Tracks and segments objects across video frames with temporal consistency
• Produces multiple valid mask hypotheses ranked by confidence scores
• Supports zero-shot transfer to new object categories without retraining
• Runs interactively for annotation workflows or in batch mode for pipelines

Architecture Overview

SAM 2 uses a Hiera image encoder (a hierarchical vision transformer) to extract multi-scale features from each frame. A prompt encoder converts user inputs (points, boxes, masks) into embeddings, and a lightweight mask decoder generates segmentation masks. For video, a memory attention module maintains a memory bank of past frames and predictions, allowing the model to propagate masks forward and backward through time. The streaming design processes one frame at a time, making it practical for long videos.

Self-Hosting & Configuration

• Install from PyPI with pip install sam-2 or clone the GitHub repository
• Download pre-trained checkpoints (tiny, small, base+, large) from the releases page
• Requires PyTorch 2.3+ and a CUDA-capable GPU for efficient inference
• Configure model size via config YAML files shipped with the repo
• Integrate into annotation tools like Label Studio or custom web UIs via the Python API

Key Features

• Unified image and video segmentation in a single model architecture
• Streaming memory design enables real-time video processing
• Four model sizes from SAM 2.1 Tiny to SAM 2.1 Large for speed-accuracy tradeoffs
• Interactive and automatic modes for both annotation and production pipelines
• Trained on SA-V, a dataset of over 50K videos with 600K masklets

Comparison with Similar Tools

• SAM 1 — image-only predecessor without video support; SAM 2 supersedes it with better image performance and adds video capabilities
• XMem / Cutie — specialized video object segmentation models; SAM 2 unifies image and video in one architecture
• GroundingDINO + SAM — combines open-vocabulary detection with segmentation; SAM 2 can be similarly composed but also handles video natively
• YOLO-Seg — instance segmentation optimized for speed; less precise masks but faster on edge devices
• Detectron2 — modular detection and segmentation framework; requires task-specific training unlike SAM 2's zero-shot capability

FAQ

Q: Can SAM 2 run without a GPU? A: CPU inference is possible but slow. A CUDA GPU with at least 6 GB VRAM is recommended for interactive use.

Q: Does SAM 2 understand object categories? A: No. SAM 2 segments objects based on spatial prompts, not semantic categories. Pair it with a classifier or detector for category labels.

Q: How do I segment objects in a long video? A: Provide a prompt on the first frame (or any frame), and SAM 2's memory mechanism will propagate the mask across subsequent frames automatically.

Q: What license is SAM 2 released under? A: SAM 2 is released under the Apache 2.0 license.

Sources

• https://github.com/facebookresearch/sam2
• https://ai.meta.com/sam2/