containerd Image Management: OCI Images and Snapshots
The containerd image management subsystem handles storing, distributing, and unpacking images based on the OCI image spec. This post analyzes the internals of Content Store, Snapshotter, image pull flow, and garbage collection.
1. OCI Image Spec
1.1 Image Structure
An OCI image consists of three core components:
OCI image structure:
1. Image Index (Fat Manifest)
- Supports multiple platforms (linux/amd64, linux/arm64, etc.)
- Points to per-platform Manifests
2. Image Manifest
- Digest of Config object
- Layer list (ordered)
- Media type information
3. Image Config
- Environment variables, entrypoint, CMD
- Layer diff ID list
- Creation history
1.2 Content Addressable Storage
Content Addressable Storage:
All objects are identified by SHA256 digest:
sha256:abc123... -> Image Index JSON
sha256:def456... -> Image Manifest JSON
sha256:789ghi... -> Image Config JSON
sha256:jkl012... -> Layer tar.gz
Benefits:
- Deduplication: identical layers stored only once
- Integrity verification: validate data via digest
- Caching: digest-based cache lookups
2. Content Store
2.1 Overview
Content Store is containerd's content-addressable storage that manages all binary data for images.
Content Store directory structure:
/var/lib/containerd/io.containerd.content.v1.content/
blobs/
sha256/
abc123... (Image Index)
def456... (Image Manifest)
789ghi... (Image Config)
jkl012... (Layer 1 tar.gz)
mno345... (Layer 2 tar.gz)
ingest/
(temporary download data)
2.2 Content Store API
Content Store key operations:
Info(digest) -> Query content metadata (size, creation time)
ReaderAt(digest) -> Read content (io.ReaderAt interface)
Writer(ref) -> Write content (atomic commit)
Delete(digest) -> Delete content
ListStatuses() -> Query in-progress writes
Abort(ref) -> Cancel in-progress write
2.3 Ingest Process
Content write (Ingest) process:
1. Create Writer (assign reference key)
|
v
2. Create temporary file in ingest/ directory
|
v
3. Stream data writes
(e.g., downloading layers from registry)
|
v
4. Digest verification
(compare expected digest with actual data hash)
|
v
5. Atomic commit
(move from ingest/ -> blobs/sha256/)
|
v
6. Clean up ingest/ on failure
3. Snapshotter
3.1 Snapshotter Overview
The Snapshotter is a plugin that manages image layers as filesystem snapshots, preparing the root filesystem for containers.
Snapshotter role:
Image layers (tar.gz)
|
v
Snapshotter converts each layer into a snapshot
|
v
Stacks snapshots to form a unified filesystem
|
v
Provides mount point to container
3.2 Snapshot Types
Snapshot types:
1. Committed
- Read-only snapshot
- Corresponds to image layers
- Sharable across multiple containers
2. Active
- Read/write snapshot
- Writable layer for a container
- Assigned to a single container
3.3 overlayfs Snapshotter
The most widely used Snapshotter:
overlayfs operation:
Layer 1 (base): /snapshots/1/fs (lowerdir)
Layer 2 (app): /snapshots/2/fs (lowerdir)
Write layer: /snapshots/3/fs (upperdir)
Work directory: /snapshots/3/work (workdir)
Mount:
mount -t overlay overlay \
-o lowerdir=/snapshots/2/fs:/snapshots/1/fs,\
upperdir=/snapshots/3/fs,\
workdir=/snapshots/3/work \
/container/rootfs
Benefits:
- Copy-on-Write: copies only on modification
- Fast container startup
- Layer sharing saves disk space
3.4 native Snapshotter
native Snapshotter:
- Stores each snapshot in an independent directory
- Fully copies parent snapshot (using hardlinks)
- Used in environments without overlayfs support
- Higher disk usage
- Simple and highly portable
3.5 devmapper Snapshotter
devmapper Snapshotter:
- Uses Linux device mapper thin provisioning
- Block-level Copy-on-Write
- Suited for high-performance workloads
- Used with Firecracker microVMs
- Complex setup (requires thin-pool pre-configuration)
Use cases:
- AWS Fargate (Firecracker)
- High-performance container environments
- Block storage-based infrastructure
3.6 Snapshotter API
Snapshotter key operations:
Stat(key) -> Query snapshot info
Prepare(key, parent) -> Create Active snapshot (writable)
View(key, parent) -> Read-only view of Committed snapshot
Commit(name, key) -> Convert Active snapshot to Committed
Mounts(key) -> Return mount info for snapshot
Remove(key) -> Delete snapshot
4. Image Pull Flow
4.1 Complete Flow
Image pull complete flow:
1. Resolve image reference
docker.io/library/nginx:latest
|
v
2. Download Image Index/Manifest
- Fetch manifest from registry
- Select manifest for target platform
|
v
3. Download Config
- Download image config JSON
- Store in Content Store
|
v
4. Download layers (parallel)
- Store each layer in Content Store
- Skip already existing layers
|
v
5. Unpack layers
- Read layers from Content Store
- Create snapshots via Snapshotter
|
v
6. Register image metadata
- Create image record in BoltDB
- Map tags to digests
4.2 Layer Download Details
Layer download:
1. Extract layer digest list from manifest
2. Check if already exists in Content Store
3. Download only missing layers from registry
4. Transfer Service manages downloads:
- Concurrent download limit (default 3)
- Progress tracking
- Retry logic
5. Each layer stored gzip-compressed in Content Store
4.3 Layer Unpacking
Layer unpacking:
1. Read layer blob from Content Store
2. Decompress gzip
3. Extract tar archive
4. Create snapshot in Snapshotter:
a. First layer: Prepare without parent
b. Apply layer contents to snapshot
c. Commit to convert to read-only
d. Next layer: Prepare with previous snapshot as parent
5. Complete final snapshot chain
Snapshot chain:
Layer 1 (committed) <- Layer 2 (committed) <- Layer 3 (committed)
5. Image Metadata
5.1 Image Record
Image metadata (BoltDB):
Image record:
- Name: "docker.io/library/nginx:latest"
- Target:
MediaType: "application/vnd.oci.image.index.v1+json"
Digest: "sha256:abc123..."
Size: 1234
- Labels:
"containerd.io/gc.ref.content.0": "sha256:def456..."
"containerd.io/gc.ref.content.1": "sha256:789ghi..."
- CreatedAt: 2026-03-20T00:00:00Z
- UpdatedAt: 2026-03-20T00:00:00Z
5.2 Querying Images
# List images with ctr
ctr -n k8s.io images list
# Detailed image info
ctr -n k8s.io images check
# Inspect image content
ctr -n k8s.io content get sha256:abc123... | jq .
6. Garbage Collection
6.1 GC Mechanism
Garbage collection operation:
1. Identify root objects:
- Image records
- Container records
- Lease records
2. Trace references (Mark):
- Image -> Manifest -> Config + Layers
- Container -> Snapshot chain
- Lease -> Protected resources
3. Delete unreferenced objects (Sweep):
- Delete unreferenced blobs from Content Store
- Delete unreferenced snapshots from Snapshotter
- Clean up orphaned metadata records
6.2 GC Labels
GC reference labels:
containerd manages GC references via labels:
Image labels:
"containerd.io/gc.ref.content.0": "sha256:..." (manifest reference)
"containerd.io/gc.ref.content.1": "sha256:..." (layer reference)
Content labels:
"containerd.io/gc.ref.content.config": "sha256:..." (config reference)
"containerd.io/gc.ref.content.l.0": "sha256:..." (layer reference)
Snapshot labels:
"containerd.io/gc.ref.snapshot.overlayfs": "sha256:..." (snapshot reference)
6.3 Lease
Lease:
- Protects in-progress operation resources from GC
- Protects downloaded layers during image pull
- Protects snapshots during container creation
- TTL-based automatic expiration
- Can be explicitly deleted after operation completes
Example:
Image pull starts -> Lease created
Layer download -> Lease protects content
Image registration complete -> Lease deleted (image record holds references)
6.4 GC Scheduling
GC triggers:
1. Thresholds on the scheduler plugin (io.containerd.gc.v1.scheduler):
- pause_threshold = 0.02
- deletion_threshold = 0
- mutation_threshold = 100
- schedule_delay = "0ms"
- startup_delay = "100ms"
2. Event-based:
- On image deletion
- On container deletion
- Explicit API call
3. Manual execution via ctr:
ctr -n k8s.io content prune references --dry
7. What You Actually Run on a Node
Everything above describes structure. Once you are on the node, what you actually hold is command output. The first wall you hit is namespaces. A containerd namespace shares nothing but the word with a Kubernetes namespace: it is a logical partition of the metadata store. Every image, snapshot and container record that kubelet obtained through the CRI plugin lives in the k8s.io namespace, while ctr with no options looks at the default namespace. That is why ctr images list on a node running dozens of pods prints an empty table, and why people conclude from it that the image vanished or that containerd lost it. Nothing vanished; you opened the wrong drawer. On a Kubernetes node, treat -n k8s.io as mandatory on every ctr invocation.
# See which namespaces exist first (NAME / LABELS)
ctr namespaces list
# Name the namespace kubelet uses
ctr -n k8s.io images list # image records
ctr -n k8s.io content ls # DIGEST / SIZE / AGE / LABELS
ctr -n k8s.io content active # REF / SIZE / AGE (ingests in flight)
ctr -n k8s.io snapshots ls # KEY / PARENT / KIND
ctr -n k8s.io snapshots tree # snapshot parent-child tree
ctr -n k8s.io snapshots usage # KEY / SIZE / INODES
ctr -n k8s.io leases ls # ID / CREATED AT / LABELS
These few lines turn every diagram above into something you can see. The LABELS column of content ls prints the reference labels from 6.2, such as containerd.io/gc.ref.content, so which blob is being held alive by what is a lookup rather than a guess. The KIND column of snapshots ls separates Committed from Active, and an Active snapshot on a node with no containers means a writable layer that was never cleaned up. snapshots usage reports size and inode count per snapshot key, which is how you catch the case where inodes run out before disk space does. content active shows downloads still sitting uncommitted in the ingest directory; in a healthy state it should only be non-empty while a pull is in progress.
For kubelet's own view, use crictl. Where ctr sees all of containerd, crictl sees only what passes through the CRI interface, so it is much closer to what kubelet believes. When the two disagree, that disagreement is itself diagnostic information.
crictl images # images the CRI knows about
crictl images --digests --no-trunc # show full digests
crictl imagefsinfo # image filesystem usage
crictl rmi --prune # remove unused images
One caution about crictl rmi. The cri-tools docs state that, because of a CRI API limitation, specifying an image by tag removes the entire image along with all of its tags, not just the one you named. If you want to drop one tag and keep the rest, the docs tell you to use a runtime-native tool such as nerdctl or ctr instead. --prune removes every image not currently in use, which on a node whose next rollout needs those same images produces a pull storm.
8. The Config Keys That Govern Image Behaviour
Check the version before you open the config file. Under version 2, used by containerd 1.x, all CRI settings sat in one block under [plugins."io.containerd.grpc.v1.cri"]. Version 3, used by containerd 2.x, splits that into a runtime side and an image side: [plugins.'io.containerd.cri.v1.runtime'] and [plugins.'io.containerd.cri.v1.images']. This version split is the most common reason a config snippet copied from a blog post or an issue thread has no effect. If the section name does not match, containerd quietly ignores it as configuration for a plugin it does not know, tells you nothing about the typo, and runs on defaults.
# containerd 1.x
version = 2
[plugins."io.containerd.grpc.v1.cri"]
snapshotter = "overlayfs"
discard_unpacked_layers = false
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
# containerd 2.x
version = 3
[plugins.'io.containerd.cri.v1.images']
snapshotter = "overlayfs"
discard_unpacked_layers = false
image_pull_progress_timeout = "5m0s"
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = "/etc/containerd/certs.d"
snapshotter defaults to overlayfs in both versions. The snapshotters the docs classify as core are overlayfs (akin to Docker/Moby's overlay2), native (akin to the vfs driver), blockfile, devmapper, btrfs, zfs and erofs; fuse-overlayfs, nydus, overlaybd and stargz are non-core plugins. btrfs and zfs require the plugin root to be mounted on that filesystem, and erofs requires the OverlayFS kernel module for active snapshots, so naming one in the config file is not enough to turn it on. Running ctr plugins ls tells you what the binary in front of you actually carries, faster than reading docs does.
discard_unpacked_layers defaults to false, and the documented description is that it allows GC to remove layers from the content store after successfully unpacking those layers to the snapshotter. Turning it on visibly reduces node disk usage because the compressed original layers are no longer retained. In exchange, the blobs are gone from the content store, so that node can no longer push or export the image, and if the snapshots ever become unusable there is no route back other than re-pulling from the registry. On pure worker nodes it is usually a win; on nodes used to build or move images it must stay off.
image_pull_progress_timeout appears in the version 3 sample config as 5m0s, and the docs describe it as the timeout for image pull progress. The name invites misreading, so treat it as a limit on stalled progress rather than a cap on the whole pull. Read it together with max_concurrent_downloads, which the docs describe as restricting the number of concurrent downloads for each image. Pulls are often slow because concurrency is low, not because the registry is.
Registry mirrors attach through config_path. The version 2 default is /etc/containerd/certs.d:/etc/docker/certs.d and the version 3 default is the empty string, so mirror configuration can silently stop being read on an upgrade to 2.x. Once a path is set, create a directory under it named after the registry host and put a hosts.toml inside. A _default directory can serve as the fallback when no other namespace matches.
# /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"
[host."https://mirror.example.com"]
capabilities = ["pull", "resolve"]
skip_verify = true
The permitted capabilities values are pull, resolve and push. For a private CA add ca, and for mutual TLS add client, in the same host block. server is the default server for this registry host namespace, and host blocks are tried before it.
9. A Worked Example: Following One Pull End to End
Let us reproduce section 4 with commands. SSH to the node and take a baseline first. crictl imagefsinfo returns image filesystem usage; the CRI API defines the mount point (fs_id.mountpoint), the bytes used for images (used_bytes) and the inodes used by images (inodes_used). The output shape can differ between releases, so check the exact field in the docs for the version you run.
# 1) baseline
crictl imagefsinfo
ctr -n k8s.io content ls | wc -l
ctr -n k8s.io snapshots ls | wc -l
# 2) pull
crictl pull registry.k8s.io/pause:3.10.2
# 3) after
crictl imagefsinfo
ctr -n k8s.io content ls
ctr -n k8s.io snapshots tree
crictl images | grep pause
If the pull really went over the network, the order is this. The manifest (or index) lands in the content store first, then the config JSON, then the layer blobs. All three appear as digests in ctr -n k8s.io content ls and are told apart by size: a few hundred bytes to a few kilobytes is a manifest or config, several megabytes and up is a layer. Unpacking then produces a new chain visible in snapshots tree, and only at the end does the image show up in crictl images. Knowing the order lets you name the stage something stopped at. Present in content but absent from snapshots means unpacking stalled; present in both but missing from crictl images means it died before the metadata record was written.
Telling a cached pull from a real one is simple. If the used bytes reported by crictl imagefsinfo barely move across the pull and the line count of content ls is unchanged, nothing came from the registry. containerd skips the download when the digest is already present, so even with imagePullPolicy Always only the manifest is checked while the layers are reused. If used bytes climb by roughly the layer size, it was a real download. When someone reports that pulls are slow, comparing those two numbers already separates a network problem from an unpacking problem.
10. Failure Cases and the Order to Diagnose Them
The most common report is a disk filling up while nothing gets deleted. containerd's GC does not run on a cron schedule; it runs on thresholds in the io.containerd.gc.v1.scheduler plugin. The documented defaults are pause_threshold = 0.02, deletion_threshold = 0, mutation_threshold = 100, schedule_delay = "0ms" and startup_delay = "100ms", and the docs explain that with the default settings the scheduler tries to keep the database unlocked 98% of the time and will not schedule itself if no deletions occurred or before 100 database writes have accumulated. In other words, what deletes images is kubelet's image garbage collection or a human, not containerd's GC. Check node disk usage, then crictl imagefsinfo, then ctr -n k8s.io snapshots usage, then kubelet's image GC thresholds, in that order.
If a node reboots mid-pull, uncommitted ingests are left behind. Content store commits are atomic, so the blobs directory is never corrupted, but the fragments in the ingest directory still occupy space. If ctr -n k8s.io content active shows a REF while no pull is in progress, this is the case. The operation that cancels an in-flight write is the Abort from 2.2, and content store cleanup is ctr -n k8s.io content prune references. Run it with the --dry flag first to see what would be removed.
Third is leases. A lease exists to protect a resource in use from GC, and ctr leases create defaults to a 24 hour expiry, with 0 meaning no expiration. When a client creates a lease and dies without deleting it, blobs that no image references stay alive indefinitely. The symptom is a short crictl images list next to a content ls full of digests, and the check is ctr -n k8s.io leases ls. An old CREATED AT means you need to find what created that lease.
Fourth is trying to re-push or export an image on a node where discard_unpacked_layers is on. The snapshots exist but the original layer blobs do not, so it fails, and reverting the setting does not bring back blobs that were already removed. Fifth is a mirror configuration that is ignored: check that the file really sits at the hosts.toml path under a registry-host-named directory in certs.d, then that config_path is actually set. Under 2.x the default is empty, so dropping the file in place accomplishes nothing on its own. Last is pull timeouts on very large layers. Here you want to know whether progress stalls rather than how long the registry round trip takes, and you tune max_concurrent_downloads together with image_pull_progress_timeout.
11. When Not to Touch This Layer Directly
On managed node groups the containerd config file is frequently regenerated during node bootstrap. A hand-edited /etc/containerd/config.toml disappears the moment the node is replaced, and worse, survives on only some nodes and creates differences you cannot reproduce. If the config must change, change the node group's bootstrap script, the launch template, or the node image itself, rather than editing on the node.
Most image problems are also solvable without touching this layer at all. If frequent pulls are the cost, lowering imagePullPolicy to IfNotPresent and pinning digests instead of tags is the better move. If registry bandwidth is the cost, attaching a mirror in hosts.toml or putting a pull-through cache inside the cluster achieves the same effect at far lower risk than swapping snapshotters. Switches like a snapshotter change or discard_unpacked_layers are hard to reverse and easily leave nodes in differing states, so keep them as the last card to play after the above.
12. References
- containerd CRI plugin config: https://github.com/containerd/containerd/blob/main/docs/cri/config.md (checked 2026-08-16)
- containerd snapshotters: https://github.com/containerd/containerd/blob/main/docs/snapshotters/README.md (checked 2026-08-16)
- containerd registry host configuration: https://github.com/containerd/containerd/blob/main/docs/hosts.md (checked 2026-08-16)
- containerd garbage collection: https://github.com/containerd/containerd/blob/main/docs/garbage-collection.md (checked 2026-08-16)
- crictl user guide: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md (checked 2026-08-16)
13. Summary
containerd image management is built on three pillars: Content Store's content-addressable storage, Snapshotter's layer management, and GC's resource cleanup. The overlayfs Snapshotter's Copy-on-Write mechanism enables fast container startup and efficient disk usage, while Lease-based GC protection ensures image operation safety.