Mastering the Segment Anything Model: Paper Analysis and Practical Guide from SAM 1 to SAM 2 to SAM 3

• 1. Overview: The Evolution of the Segment Anything Model
• 2. SAM 1: Segment Anything (2023)
  • 2.1 Paper Information
  • 2.2 Three Key Contributions
  • 2.3 Architecture Details
    • Image Encoder
    • Prompt Encoder
    • Mask Decoder
  • 2.4 SA-1B Dataset
    • Data Engine — 3-Phase Construction Process
  • 2.5 Key Innovations
  • 2.6 Installation and Usage
• 3. SAM 2: Segment Anything in Images and Videos (2024)
  • 3.1 Paper Information
  • 3.2 What Changed from SAM 1
  • 3.3 Architecture Details
    • Hiera Image Encoder
    • Memory Mechanism (Key Innovation)
    • Occlusion Prediction Head
  • 3.4 SA-V Dataset
  • 3.5 Model Variants and Performance
  • 3.6 Installation and Usage
• 4. SAM 3: Segment Anything with Concepts (2025)
  • 4.1 Paper Information
  • 4.2 Paradigm Shift: From Where to What
  • 4.3 Architecture Details
    • Key Innovation: Presence Head
    • Decoupled Detector-Tracker Design
    • Training Pipeline (4 Stages)
  • 4.4 SA-Co Dataset
    • Training Data
    • Evaluation Benchmarks
    • Additional Public Datasets
  • 4.5 Performance Benchmarks
    • Image Segmentation (Zero-Shot)
    • Video Segmentation
    • Exemplar Utilization Effect
    • Speed and Efficiency
  • 4.6 Installation and Usage
    • Official Repository
    • Ultralytics Integration
• 5. Comprehensive Comparison of All Three Models
  • 5.1 Architecture Comparison
  • 5.2 Dataset Comparison
  • 5.3 Performance Evolution
• 6. Practical Use Cases
  • 6.1 When SAM 1 Is the Right Choice
  • 6.2 When SAM 2 Is the Right Choice
  • 6.3 When SAM 3 Is the Right Choice
  • 6.4 Real-World Deployment Examples (SAM 3)
• 7. Study Roadmap
  • 7.1 Prerequisites
  • 7.2 Recommended Paper Reading Order
  • 7.3 Recommended Hands-On Exercises
  • 7.4 Related Follow-Up Research
• 8. References
  • Core Papers
  • Prerequisite Papers
  • GitHub Repositories
  • Tutorials and Guides
  • Project Pages and Datasets
• Quiz

1. Overview: The Evolution of the Segment Anything Model

Segment Anything Model (SAM) is a series of foundation models for image and video segmentation published by Meta AI Research. Just as GPT established the "prompt" paradigm in NLP, SAM introduced a new paradigm called Promptable Segmentation to computer vision segmentation.

The core evolutionary direction of the three models can be summarized in one line:

SAM 1: "Where to segment?" → SAM 2: "Where — even in video?" → SAM 3: "What to segment?"

2. SAM 1: Segment Anything (2023)

2.1 Paper Information

• Title: Segment Anything
• Authors: Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al. (Meta AI Research)
• Published: April 2023 (arXiv), accepted at ICCV 2023
• Paper: arxiv.org/abs/2304.02643
• GitHub: github.com/facebookresearch/segment-anything
• License: Apache 2.0

2.2 Three Key Contributions

SAM 1 simultaneously introduced three things:

1. A new task — Promptable Segmentation: Given any prompt (point, box, mask, or text), the model returns a valid segmentation mask
2. A new model — SAM: A foundation model for prompt-based segmentation
3. A new dataset — SA-1B: The largest segmentation dataset ever, containing 1.1 billion masks from 11 million images

2.3 Architecture Details

SAM's architecture is decomposed into three components. The core design principle is to run the heavy image encoder only once and repeatedly call the lightweight prompt encoder and mask decoder in real time.

┌─────────────────────────────────────────────────────────┐
│                      SAM Architecture                    │
│                                                          │
│  ┌──────────────┐   ┌────────────────┐                   │
│  │ Image Encoder │   │ Prompt Encoder │                   │
│  │  (ViT-H/L/B) │   │ (Point/Box/    │                   │
│  │  MAE Pretrain │   │  Mask/Text)    │                   │
│  └──────┬───────┘   └───────┬────────┘                   │
│         │                   │                            │
│         └─────────┬─────────┘                            │
│                   ▼                                      │
│          ┌────────────────┐                              │
│          │  Mask Decoder   │                              │
│          │ (Transformer    │                              │
│          │  2-layer)       │──→ 3 masks + IoU scores      │
│          └────────────────┘                              │
└─────────────────────────────────────────────────────────┘

Image Encoder

• Vision Transformer pretrained with MAE (Masked Autoencoder)
• Input resolution: 1024×1024
• Output: 64×64 image embedding (256-dim)
• Runs only once per image — reused for each prompt afterwards

Prompt Encoder

Sparse prompts (points, boxes, text):

• Points/boxes → Positional Encoding + learned type embeddings (foreground point vs. background point)
• Text → processed via CLIP text encoder

Dense prompts (masks):

• Embedded via convolution layers and element-wise summed with the image embedding

Mask Decoder

• Modified Transformer Decoder (2 layers)
• Embedding dimension: 256, MLP inner dimension: 2048
• Ambiguity-aware output: predicts 3 candidate masks simultaneously for a single prompt
  • Whole, Part, and Sub-part levels
  • Each mask is assigned an IoU confidence score
• ~50ms inference per prompt on GPU — enabling real-time interaction

2.4 SA-1B Dataset

Data Engine — 3-Phase Construction Process

Phase 1: Assisted-Manual        Phase 2: Semi-Automatic       Phase 3: Fully Automatic
─────────────────────          ──────────────────────         ──────────────────────
• SAM + human annotators        • SAM auto-suggests masks     • 32×32 grid points auto-generated
• Browser-based tool            • Humans add missed objects   • NMS + confidence filtering
• 120K images / 4.3M masks     • 180K images / 5.9M masks    • Majority of 1.1B masks

Quality verification: Human audit of 500 images (~50,000 masks) showed 94% achieving IoU > 0.90

2.5 Key Innovations

Zero-Shot Transfer: Instantly applicable to new domains without fine-tuning

• Edge detection (BSDS500 ODS: 0.768)
• Object proposal generation (LVIS AR@1000: 59.3)
• Medical imaging, satellite imagery, underwater photography, microscopy images, etc.

Benchmark performance:

• Zero-shot evaluation across 23 datasets: outperformed existing SOTA (RITM) on 16
• With oracle selection: outperformed RITM on all 23
• No performance disparity across demographic groups such as race/gender (fairness validation)

2.6 Installation and Usage

# Installation
pip install git+https://github.com/facebookresearch/segment-anything.git

# Optional dependencies
pip install opencv-python pycocotools matplotlib onnxruntime onnx

Checkpoint download:

Prompt-based segmentation:

from segment_anything import SamPredictor, sam_model_registry
import numpy as np

# Load model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

# Create predictor
predictor = SamPredictor(sam)

# Set image (image embedding computed once)
predictor.set_image(image)  # numpy array (H, W, 3) RGB

# Predict with point prompt
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375]]),  # (N, 2) coordinates
    point_labels=np.array([1]),            # 1=foreground, 0=background
    multimask_output=True,                 # Return 3 candidate masks
)

Automatic mask generation (Segment Everything):

from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
sam.to(device="cuda")

mask_generator = SamAutomaticMaskGenerator(sam)
masks = mask_generator.generate(image)
# Each mask: segmentation, area, bbox, predicted_iou, stability_score

3. SAM 2: Segment Anything in Images and Videos (2024)

3.1 Paper Information

• Title: SAM 2: Segment Anything in Images and Videos
• Authors: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu et al. (15 authors, Meta AI Research)
• Published: August 2024 (arXiv), revised October 2024
• Paper: arxiv.org/abs/2408.00714
• GitHub: github.com/facebookresearch/sam2
• Project page: ai.meta.com/sam2
• License: Apache 2.0

3.2 What Changed from SAM 1

The core question behind SAM 2: "Can we extend image-only segmentation to video?"

3.3 Architecture Details

┌────────────────────────────────────────────────────────────────────┐
│                    SAM 2 Streaming Architecture                    │
│                                                                    │
│  Frame t-2      Frame t-1      Frame t (current)                   │
│     │            │            │                                    │
│     ▼            ▼            ▼                                    │
│  ┌──────┐    ┌──────┐    ┌──────────────┐                          │
│  │Hiera │    │Hiera │    │ Hiera Image  │                          │
│  │Enc.  │    │Enc.  │    │   Encoder    │                          │
│  └──┬───┘    └──┬───┘    └──────┬───────┘                          │
│     │           │               │                                  │
│     ▼           ▼               ▼                                  │
│  ┌──────────────────────────────────────────┐                      │
│  │            Memory Bank                    │                      │
│  │  ┌──────────────┐  ┌───────────────────┐ │                      │
│  │  │ FIFO Queue   │  │ Prompted Frames   │ │                      │
│  │  │ (N recent    │  │ (M prompted       │ │                      │
│  │  │  frames)     │  │  frames)          │ │                      │
│  │  └──────────────┘  └───────────────────┘ │                      │
│  │  ┌──────────────────────────────────────┐│                      │
│  │  │     Object Pointers (256-dim)        ││                      │
│  │  └──────────────────────────────────────┘│                      │
│  └──────────────────┬───────────────────────┘                      │
│                     ▼                                              │
│  ┌──────────────────────────────┐   ┌───────────────┐              │
│  │    Memory Attention Module    │←──│Prompt Encoder │              │
│  │ (L Transformer Blocks:       │   │(Point/Box/    │              │
│  │  Self-Attn + Cross-Attn)     │   │ Mask)         │              │
│  └──────────────┬───────────────┘   └───────────────┘              │
│                 ▼                                                   │
│  ┌──────────────────────────┐                                      │
│  │      Mask Decoder        │──→ Mask + IoU + Occlusion Score      │
│  │ (+ Skip Connections)     │                                      │
│  └──────────────────────────┘                                      │
└────────────────────────────────────────────────────────────────────┘

Hiera Image Encoder

SAM 1's ViT was replaced with Hiera (hierarchical ViT).

• Stages 3+4 (stride 16, 32) → fused via FPN → image embedding (input to Memory Attention)
• Stages 1+2 (stride 4, 8) → skip connections to the mask decoder's upsampling layers
• Uses windowed absolute positional embedding

Memory Mechanism (Key Innovation)

Memory Encoder: After processing each frame, frame features and predicted masks are stored in the Memory Bank (projected to 64-dim)

Memory Bank structure:

• FIFO Queue: The most recent N non-prompted frames — maintains temporal context
• Prompted Frames: Up to M prompted frames — preserves user guidance
• Object Pointers: 256-dim vectors — high-level semantic information about tracked objects

Memory Attention Module: L Transformer blocks

• Self-Attention: within the current frame features
• Cross-Attention: interaction with past frames + object pointers from the Memory Bank

Occlusion Prediction Head

Handles situations in video where objects become occluded or leave the frame:

• Outputs a visibility score per frame
• Suppresses noisy predictions for occluded objects
• On reappearance → immediate re-identification via the Memory Bank

3.4 SA-V Dataset

3-Phase Data Engine:

3.5 Model Variants and Performance

SAM 2.1 (released September 2024, latest recommended version):

Key benchmarks:

• Image segmentation: 6x faster than SAM 1 while also improving accuracy
• Interaction efficiency: 3x fewer user interactions to achieve the same quality

3.6 Installation and Usage

# Installation
git clone https://github.com/facebookresearch/sam2.git && cd sam2
pip install -e .

# Download checkpoints
cd checkpoints && ./download_ckpts.sh && cd ..

Image segmentation:

import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint))

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(image)
    masks, _, _ = predictor.predict(
        point_coords=np.array([[500, 375]]),
        point_labels=np.array([1]),
    )

Video segmentation:

import torch
from sam2.build_sam import build_sam2_video_predictor

checkpoint = "./checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
predictor = build_sam2_video_predictor(model_cfg, checkpoint)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(video_path)

    # Add prompt to a specific frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=np.array([[500, 375]]),
        labels=np.array([1]),
    )

    # Propagate through entire video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        # Process masks for each frame
        process_masks(frame_idx, masks)

Hugging Face integration:

from sam2.sam2_image_predictor import SAM2ImagePredictor
predictor = SAM2ImagePredictor.from_pretrained("facebook/sam2-hiera-large")

from sam2.sam2_video_predictor import SAM2VideoPredictor
predictor = SAM2VideoPredictor.from_pretrained("facebook/sam2-hiera-large")

4. SAM 3: Segment Anything with Concepts (2025)

4.1 Paper Information

• Title: SAM 3: Segment Anything with Concepts
• Authors: Nicolas Carion, Laura Gustafson, Yuan-Ting Hu et al. (35 authors, Meta AI Research)
• Published: November 20, 2025 (arXiv), accepted at ICLR 2026
• Paper: arxiv.org/abs/2511.16719
• GitHub: github.com/facebookresearch/sam3
• Project page: ai.meta.com/sam3
• License: Apache 2.0

4.2 Paradigm Shift: From Where to What

SAM 1 and SAM 2 relied on geometric prompts (points, boxes) that specify "where" to segment. SAM 3 asks a fundamentally different question: "What" to segment?

SAM 1: Point/Box → "Isolate the object at this location"
SAM 2: Point/Box + Video Tracking → "Track this object through the video"
SAM 3: Text/Image Exemplar → "Find everything matching this concept"

Promptable Concept Segmentation (PCS) — the new task proposed by SAM 3:

• Text prompts: Natural language noun phrases (e.g., "yellow school bus")
• Image exemplars: Visual example images
• Combined prompts: Text + exemplar used simultaneously
• Legacy prompts: Points, boxes, masks (backward compatible with SAM 2)

4.3 Architecture Details

Total parameters: 848M (~3.4 GB)

┌───────────────────────────────────────────────────────────────────────┐
│                       SAM 3 Architecture                              │
│                                                                       │
│  ┌────────────┐  ┌────────────┐  ┌──────────────────┐                 │
│  │ Text       │  │ Exemplar   │  │ Visual Prompts   │                 │
│  │ Encoder    │  │ Encoder    │  │ (Point/Box/Mask) │                 │
│  │ "school    │  │ [image]    │  │                  │                 │
│  │  bus"      │  │            │  │                  │                 │
│  └─────┬──────┘  └─────┬──────┘  └────────┬─────────┘                 │
│        └────────────┬───┘                  │                          │
│                     ▼                      │                          │
│  ┌──────────────────────────────┐          │                          │
│  │   Meta Perception Encoder    │          │                          │
│  │   (Vision-Language joint     │          │                          │
│  │    embedding space)          │          │                          │
│  └──────────────┬───────────────┘          │                          │
│                 │                          │                          │
│        ┌────────┴────────┐                 │                          │
│        ▼                 ▼                 │                          │
│  ┌───────────┐    ┌─────────────────┐      │                          │
│  │ Presence  │    │  Fusion Encoder │      │                          │
│  │   Head    │    │  (Conditional   │      │                          │
│  │ "yes/no"  │    │   features)     │      │                          │
│  └───────────┘    └────────┬────────┘      │                          │
│                            │               │                          │
│                            ▼               │                          │
│               ┌──────────────────┐         │                          │
│               │  DETR Detector   │←────────┘                          │
│               │  (Transformer-   │                                    │
│               │   based detect.) │                                    │
│               └────────┬─────────┘                                    │
│                        │                                              │
│                        ▼                                              │
│  ┌──────────────────────────────────────┐                             │
│  │    SAM 2-inspired Tracker            │                             │
│  │    (Memory Bank + temporal           │                             │
│  │     consistency)                     │                             │
│  └──────────────────┬───────────────────┘                             │
│                     ▼                                                 │
│              Masks + Boxes + Scores                                   │
└───────────────────────────────────────────────────────────────────────┘

Key Innovation: Presence Head

The Presence Head is SAM 3's most important architectural innovation. It decouples recognition from localization.

Previous approach: Detector simultaneously "finds" and "judges existence" → optimization conflict
SAM 3:            Presence Head first judges "does it exist?" → Detector focuses solely on "finding"

Impact:

Decoupled Detector-Tracker Design

Unlike SAM 2's monolithic architecture, SAM 3 separates detection and tracking into independent modules:

• Shared: backbone (Perception Encoder)
• Separate: decoder heads (detection vs. tracking)
• Advantage: reduced inter-task interference, independent scaling

Training Pipeline (4 Stages)

4.4 SA-Co Dataset

SAM 3 was trained on SA-Co (Segment Anything with Concepts), the largest and most diverse segmentation dataset ever.

Training Data

Total ontology: 22 million entities (17 top-level categories, 72 subcategories)

Evaluation Benchmarks

• 214,000 unique phrases, 126,000 images/videos — over 50x more diverse concepts than existing COCO/LVIS

Additional Public Datasets

• SA-FARI: 10,000+ wildlife camera trap videos (100+ species annotated)
• FathomNet: Underwater segmentation benchmark for marine imagery

4.5 Performance Benchmarks

Image Segmentation (Zero-Shot)

Video Segmentation

Exemplar Utilization Effect

Speed and Efficiency

• Inference speed: ~30ms per image on H200 GPU (including 100+ object detection)
• Model size: 848M parameters (~3.4 GB)
• Runnable on 16 GB GPUs
• vs. human performance: achieves 88% of the human lower bound on SA-Co/Gold

4.6 Installation and Usage

Official Repository

# Create environment
conda create -n sam3 python=3.12
conda activate sam3

# Install PyTorch
pip install torch==2.7.0 torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126

# Install SAM 3
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

# Optional: for notebooks or development
pip install -e ".[notebooks]"
pip install -e ".[train,dev]"

Image segmentation (text prompt):

from PIL import Image
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

model = build_sam3_image_model()
processor = Sam3Processor(model)
image = Image.open("path/to/image.jpg")

# Concept segmentation with text prompt
inference_state = processor.set_image(image)
output = processor.set_text_prompt(
    state=inference_state,
    prompt="yellow school bus"
)
masks, boxes, scores = output["masks"], output["boxes"], output["scores"]

Video tracking:

from sam3.model_builder import build_sam3_video_predictor

video_predictor = build_sam3_video_predictor()
response = video_predictor.handle_request(
    request=dict(
        type="start_session",
        resource_path="path/to/video.mp4"
    )
)

Ultralytics Integration

pip install -U ultralytics

from ultralytics.models.sam import SAM3SemanticPredictor

overrides = dict(
    conf=0.25, task="segment", mode="predict",
    model="sam3.pt", half=True
)
predictor = SAM3SemanticPredictor(overrides=overrides)
predictor.set_image("path/to/image.jpg")

# Segment multiple concepts simultaneously
results = predictor(text=["person", "bus", "glasses"])

5. Comprehensive Comparison of All Three Models

5.1 Architecture Comparison

5.2 Dataset Comparison

5.3 Performance Evolution

6. Practical Use Cases

6.1 When SAM 1 Is the Right Choice

• When fast segmentation on single images is needed
• Building interactive annotation tools
• When simple integration into existing pipelines is needed (most mature ecosystem)

6.2 When SAM 2 Is the Right Choice

• When video object tracking is needed
• Real-time video stream analysis
• Scenarios where occlusion handling is critical
• When faster speeds than SAM 1 are needed for images as well

6.3 When SAM 3 Is the Right Choice

• When segmenting via natural language search ("find all the cars")
• Open-vocabulary object detection + segmentation
• Visual exemplar-based few-shot segmentation
• Large-scale automatic labeling of images/videos

6.4 Real-World Deployment Examples (SAM 3)

• Instagram Edits app: Applying dynamic effects to specific people/objects in videos
• Facebook Marketplace "View in Room": AR furniture placement (SAM 3 + SAM 3D)
• Wildlife conservation: Camera trap monitoring based on the SA-FARI dataset

7. Study Roadmap

A step-by-step learning guide for deeply understanding and utilizing the SAM series.

7.1 Prerequisites

7.2 Recommended Paper Reading Order

1. SAM 1 paper (2023) → Establish foundational concepts
   ↓
2. SAM 2 paper (2024) → Understand video extension
   ↓
3. SAM 3 paper (2025) → Understand concept awareness
   ↓
4. Hands-on notebooks from each GitHub repository

7.3 Recommended Hands-On Exercises

7.4 Related Follow-Up Research

8. References

Core Papers

1. Kirillov, A., Mintun, E., Ravi, N., et al. (2023). "Segment Anything". ICCV 2023. arxiv.org/abs/2304.02643
2. Ravi, N., Gabeur, V., Hu, Y.-T., et al. (2024). "SAM 2: Segment Anything in Images and Videos". arxiv.org/abs/2408.00714
3. Carion, N., Gustafson, L., Hu, Y.-T., et al. (2025). "SAM 3: Segment Anything with Concepts". ICLR 2026. arxiv.org/abs/2511.16719

Prerequisite Papers

4. Dosovitskiy, A., et al. (2021). "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale". ICLR 2021. arxiv.org/abs/2010.11929
5. He, K., et al. (2022). "Masked Autoencoders Are Scalable Vision Learners". CVPR 2022. arxiv.org/abs/2111.06377
6. Carion, N., et al. (2020). "End-to-End Object Detection with Transformers (DETR)". ECCV 2020. arxiv.org/abs/2005.12872
7. Radford, A., et al. (2021). "Learning Transferable Visual Models From Natural Language Supervision (CLIP)". ICML 2021. arxiv.org/abs/2103.00020
8. Ryali, C., et al. (2023). "Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles". ICML 2023. arxiv.org/abs/2306.00989

GitHub Repositories

9. github.com/facebookresearch/segment-anything — SAM 1
10. github.com/facebookresearch/sam2 — SAM 2
11. github.com/facebookresearch/sam3 — SAM 3

Tutorials and Guides

12. Ultralytics SAM Documentation — docs.ultralytics.com/models/sam
13. Ultralytics SAM 2 Documentation — docs.ultralytics.com/models/sam-2
14. Ultralytics SAM 3 Documentation — docs.ultralytics.com/models/sam-3
15. Encord Blog: Segment Anything Model Explained — encord.com/blog/segment-anything-model-explained
16. Roboflow: What is SAM 3 — blog.roboflow.com/what-is-sam3
17. Meta AI Blog: Introducing SAM 3 — ai.meta.com/blog/segment-anything-model-3

Project Pages and Datasets

18. Meta AI SAM 2 Project — ai.meta.com/sam2
19. Meta AI SAM 3 Project — ai.meta.com/sam3
20. SA-1B Dataset — ai.meta.com/datasets/segment-anything
21. SA-V Dataset — ai.meta.com/datasets/segment-anything-video

Quiz