containerd Container Lifecycle Management
This post analyzes the complete lifecycle of containers in containerd, from creation to termination. We examine the separation of container metadata and execution processes (Tasks), process management through shims, and integration of various runtime classes.
1. Separation of Container and Task
1.1 Core Concepts
containerd separates container metadata from execution state:
Container (metadata):
- ID, image reference, snapshot key
- OCI runtime spec
- Labels, extension data
- Persisted in BoltDB
Task (execution state):
- Actually running process
- PID, state (created/running/stopped)
- stdin/stdout/stderr
- Managed by shim process
1.2 Benefits of Separation
Benefits of separated design:
1. Container metadata can exist without a Task
- Create container and start later
- Preserve metadata of stopped containers
2. Independent of containerd restarts
- Tasks are managed by shim, surviving containerd restarts
- Reconnect to existing shims after restart
3. Support for multiple runtimes
- Container object is runtime-agnostic
- Runtime selected at Task creation time
2. Container Creation
2.1 Creation Process
Container creation flow:
1. Generate OCI spec from image
|
v
2. Prepare snapshot
- Add Active snapshot to image snapshot chain
- Writable layer for the container
|
v
3. Store container metadata
- Create Container record in BoltDB
- Store ID, image, snapshot, runtime, spec
|
v
4. Return container object
(process not yet started)
2.2 OCI Runtime Spec
containerd generates an OCI runtime spec to define the container execution environment:
OCI runtime spec key sections:
ociVersion: "1.0.2"
process:
terminal: false
user: uid=0, gid=0
args: ["/bin/sh"]
env: ["PATH=/usr/local/sbin:..."]
cwd: "/"
capabilities: ...
rlimits: ...
root:
path: "rootfs"
readonly: false
hostname: "container-abc"
mounts:
- destination: "/proc"
type: "proc"
source: "proc"
- destination: "/dev"
type: "tmpfs"
source: "tmpfs"
linux:
namespaces:
- type: "pid"
- type: "network"
- type: "ipc"
- type: "uts"
- type: "mount"
resources:
memory:
limit: 536870912
cpu:
shares: 1024
quota: 100000
period: 100000
cgroupsPath: "/kubelet/pod-abc/container-xyz"
2.3 Spec Generators (Spec Opts)
containerd spec generation pattern:
Spec Opts are function chains that incrementally build the OCI spec:
WithImageConfig(image) -> Apply image CMD, ENV, WORKDIR
WithHostNamespace(ns) -> Share host namespace
WithMemoryLimit(limit) -> Set memory limit
WithCPUs(cpus) -> Set CPU limit
WithMounts(mounts) -> Add mount points
WithProcessArgs(args) -> Set process arguments
WithRootfsPropagation(p) -> Set rootfs mount propagation
WithSeccompProfile(p) -> Apply Seccomp profile
WithApparmorProfile(p) -> Apply AppArmor profile
3. Task Execution
3.1 Task Creation
Task creation flow:
1. Check container's runtime type
(e.g., io.containerd.runc.v2)
|
v
2. Execute shim binary
(containerd-shim-runc-v2 start)
|
v
3. Shim returns ttrpc socket address
|
v
4. containerd sends Create request to shim
- Pass OCI spec
- Pass bundle path
|
v
5. Shim executes runc create
- Create namespaces
- Configure cgroups
- Mount rootfs
- Create process (not yet started)
|
v
6. Task state: Created
3.2 Task Start
Task start:
1. containerd sends Start request to shim
|
v
2. Shim executes runc start
- Start container process init
- Synchronize via exec.fifo
|
v
3. Task state: Running
- PID assigned
- stdin/stdout/stderr connected
3.3 Task State Transitions
Task state machine:
Created
|
| Start()
v
Running
|
+-- Kill(signal) -> Send signal
|
+-- Pause() -> Paused
| |
| +-- Resume() -> Running
|
+-- Process exits -> Stopped
|
v
Stopped
|
| Delete()
v
(deleted)
3.4 Exec (Additional Processes)
Exec operation:
Add a new process to an already running container:
1. Create ExecProcess
- Define new process spec (args, env, user)
- Assign execID
|
v
2. Send Exec request to shim
|
v
3. Execute runc exec
- Enter existing container namespaces
- Start new process
|
v
4. Independently manage stdin/stdout/stderr
Use cases: kubectl exec, docker exec
4. Shim Lifecycle
4.1 Shim Start
Shim start process:
1. containerd fork/execs shim binary
containerd-shim-runc-v2 -namespace k8s.io \
-id container-abc \
-address /run/containerd/containerd.sock \
start
|
v
2. Shim daemonizes itself
- Detach from parent process (setsid)
- Run independently of containerd
|
v
3. Create ttrpc Unix socket
/run/containerd/s/abc123...
|
v
4. Output socket address to stdout
containerd reads this address to connect
4.2 Shim Responsibilities
Shim key responsibilities:
1. Process management:
- Act as parent of container process
- Collect exit status via wait4()
- Detect and report OOM events
2. I/O management:
- Manage stdin/stdout/stderr FIFOs
- Connect to log drivers
- Copy I/O (containerProcess <-> FIFO)
3. Communication with containerd:
- Receive commands via ttrpc
- Report events (TaskExit, etc.)
- Respond to status queries
4. containerd restart resilience:
- Continue running when containerd restarts
- Restarted containerd reconnects to existing shim
- State recovery
4.3 Shim Shutdown
Shim shutdown:
1. Receive Task Delete request
|
v
2. Clean up container resources
- Delete cgroups
- Clean up namespaces
- Unmount rootfs
|
v
3. Close ttrpc socket
|
v
4. Shim process exits
5. Checkpoint/Restore
5.1 Checkpoint
Checkpoint operation:
Save running container state as a snapshot:
1. Invoke CRIU (Checkpoint/Restore in Userspace)
|
v
2. Dump process memory
- Save memory pages
- Save file descriptor state
- Save network connection state
|
v
3. Create checkpoint image
- CRIU image file set
- Stored alongside container spec
|
v
4. Optionally stop the container
Use cases:
- Live migration
- Fast start (restore from pre-warmed state)
- Debugging (capture state at specific point)
5.2 Restore
Restore operation:
1. Load checkpoint image
|
v
2. Prepare new container environment
- Create namespaces
- Mount rootfs
|
v
3. Execute CRIU restore
- Restore memory pages
- Restore process state
- Reconnect file descriptors
|
v
4. Resume process execution
6. Runtime Classes
6.1 Multiple Runtime Support
containerd supports various runtimes through the shim interface:
Runtime class comparison:
+----------+------------+-----------+----------+---------------+
| Runtime | Isolation | Overhead | Startup | Compatibility |
+----------+------------+-----------+----------+---------------+
| runc | Namespace | Minimal | Fast | Best |
| kata | Light VM | Medium | Medium | High |
| gVisor | User kernel| Low | Fast | Medium |
| Wasm | Wasm sandbox| Minimal | Very fast| Limited |
+----------+------------+-----------+----------+---------------+
6.2 runc
runc:
- Default OCI runtime
- Linux namespace and cgroup-based isolation
- Uses host kernel directly
- Lowest overhead
- Suitable for all Linux container workloads
shim: containerd-shim-runc-v2
config.toml:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
6.3 Kata Containers
Kata Containers:
- Runs containers inside lightweight VMs
- Uses QEMU/Cloud-Hypervisor/Firecracker
- Strong isolation with separate guest kernel
- Suited for multi-tenant environments
- VM overhead exists
shim: containerd-shim-kata-v2
config.toml:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
6.4 gVisor
gVisor (runsc):
- User-space kernel (Sentry)
- Intercepts and reimplements system calls
- Reduces host kernel attack surface
- Operates via ptrace or KVM
- Some system calls unsupported
shim: containerd-shim-runsc-v1
config.toml:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
runtime_type = "io.containerd.runsc.v1"
6.5 WebAssembly (Wasm)
Wasm runtime:
- Runs WebAssembly binaries as containers
- Uses Wasmtime, WasmEdge, etc.
- Very fast startup (millisecond range)
- Minimal memory usage
- Portable binaries
- Limited system access (WASI)
shim: containerd-shim-wasm
config.toml:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasm]
runtime_type = "io.containerd.wasm.v1"
6.6 RuntimeClass Selection
Kubernetes RuntimeClass integration:
1. Define RuntimeClass resource:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: kata
handler: kata
2. Specify RuntimeClass in Pod:
spec:
runtimeClassName: kata
containers:
- name: app
image: nginx
3. containerd selects runtime matching handler:
handler "kata" -> containerd.runtimes.kata config
-> execute containerd-shim-kata-v2
7. Seeing Container and Task Directly with ctr and crictl
The separation described in section 1 is not a conceptual device — it shows up literally in command output. There are two separate listings, and the fact that their line counts can differ is the proof.
The first trap is the namespace. containerd stores metadata in namespaces isolated per client, and everything kubelet creates lands in the k8s.io namespace. Drop the -n k8s.io flag and you are looking at the default namespace, the listing comes back empty, and you conclude the container is gone. When investigating on a node, that flag is not optional.
# Containers (metadata) — columns are CONTAINER, IMAGE, RUNTIME
ctr -n k8s.io containers list
# Tasks (running processes) — columns are TASK, PID, STATUS
ctr -n k8s.io tasks list
# What plugins and snapshotters this build actually carries
ctr plugins ls
Put the two listings side by side and the separation becomes visible. An entry present in containers but absent from tasks is a container that is metadata only, with no process. Even a healthy node shows this state briefly: a container has been created but not started yet, or the process has exited but has not been deleted. The problem is when the state persists. That is a signal that post-exit cleanup is stuck, and the next place to look is the diagnosis order in section 9.
To go deep on one container, use info. This is where the OCI runtime spec from section 2 shows up filled in with real values.
# The full container record (image, snapshot key, runtime, labels)
ctr -n k8s.io containers info CONTAINER_ID
# Only the OCI spec
ctr -n k8s.io containers info CONTAINER_ID --spec
# Processes inside the task, and its resource metrics
ctr -n k8s.io tasks ps CONTAINER_ID
ctr -n k8s.io tasks metrics CONTAINER_ID
There are specific values to check in the --spec output: whether the memory limit and CPU quota under linux.resources match what the Pod spec asked for, which namespaces appear under linux.namespaces and which are shared with the sandbox, and whether cgroupsPath is the path you expect. When the Pod spec and this output disagree, something intervened during translation — usually kubelet, or the runtime handler configuration from section 6.
Looking at the same container through crictl changes the perspective. ctr shows containerd's own objects; crictl shows the model CRI defines, which is pods and containers. When you are checking something in a Kubernetes context, crictl is the right tool. Which Pod a container belongs to, and which restart attempt this is, are facts ctr simply does not have.
crictl ps # running containers
crictl ps -a # including exited ones
crictl inspect CONTAINER_ID
crictl stats
8. Worked Example: Restarting containerd Leaves Containers Running
The fastest way to understand what the shim daemonization in section 4 buys you is to reproduce it. On a staging node, walk this sequence exactly.
# 1) Baseline — note the task list and one of the PIDs
ctr -n k8s.io tasks list
# 2) Count the shim processes currently running
pgrep -c containerd-shim-runc-v2
# 3) Restart containerd only (leave kubelet alone)
systemctl restart containerd
# 4) Check again — the tasks are unchanged, and so are the PIDs
ctr -n k8s.io tasks list
pgrep -c containerd-shim-runc-v2
In step 3 the containerd process really does die and come back. Yet step 4 prints what step 1 printed. That the task PIDs did not change is the whole point. The parent of the container process is the shim, not containerd, and the shim daemonizes itself into a separate session, so containerd disappearing does not touch it. The restarted containerd re-discovers each shim's ttrpc socket address and reconnects, and from that moment status queries and commands resume.
The lesson from this experiment is not reassurance but an order of investigation. When a report comes in that a container died, restarting containerd usually fixes nothing — the shim is what actually holds the container, so restarting containerd only replaces the management plane. The same property also creates a failure mode of its own: if reattachment fails, the container process keeps running perfectly while containerd cannot manage it. The symptom is a mismatch where ctr tasks list shows nothing but pgrep containerd-shim-runc-v2 still finds processes, and at that point you check whether the socket paths still exist and read the containerd log for reattachment errors.
9. Failure Modes and the Order to Check Them
A task stuck in Stopped that will not delete is the most frequently reported symptom. crictl ps -a shows it as Exited, it does not go away, and the same Pod keeps restarting. Recall the state machine from section 3: Stopped is not the terminus, it is the state waiting for Delete. Deletion usually stalls because a cleanup step is blocked — a cgroup that will not be removed, or a rootfs unmount that fails. The order is task state, then leftover mounts, then the cleanup errors in the containerd log. If you must force it, ctr -n k8s.io tasks delete --force CONTAINER_ID kills the process and proceeds with deletion. But that changes state behind kubelet's back, so keep it as a last resort after the diagnosis is done.
Second is the orphaned shim. It is left behind when containerd is killed rather than shut down cleanly, and the symptom is the mismatch described in section 8. If the node's shim process count is noticeably higher than the actual container count, suspect this. One shim holds little on its own, but they accumulate along with cgroups and mounts, and eventually new container creation fails. Draining and replacing the node is almost always safer than cleaning up by hand.
Third is a cgroup driver mismatch. Containers start but restart unpredictably, resource limits do not take effect as intended, and replacing the node reproduces it. Per the containerd docs, the runc option SystemdCgroup defaults to false in both config version 2 and version 3. Meanwhile kubelet on most recent distributions uses the systemd driver. Leaving the default in place is therefore an easy way to end up mismatched. Check the value in the config file against kubelet's cgroup driver setting, keeping in mind that the config path differs by config version as covered in 6.2.
# Dump this build's own defaults and compare against the current file
containerd config default > /tmp/containerd-default.toml
grep -n -i "systemdcgroup\|runtime_type\|version" /tmp/containerd-default.toml | head
Fourth is a runtime handler that does not exist. The RuntimeClass from 6.6 passes Kubernetes-side validation, but the moment kubelet passes that handler name over CRI, containerd cannot find a matching runtimes entry and sandbox creation fails. The RuntimeClass handler and the key under runtimes in the config must match character for character, and the shim binary named by that entry's runtime_type must be on PATH. If you installed Kata and Pods will not start, checking those three in order is the fastest route.
Fifth is a task stranded in Paused. As the state machine in 3.3 shows, a task can be stopped with Pause and only Resume releases it. The processes are frozen by the freezer cgroup, so they are alive and the PID is unchanged, but nothing happens. From the application's point of view it looks completely hung, CPU usage is zero, and logs stop. This state is left behind when a person ran ctr tasks pause or a checkpoint tool failed halfway. Check the STATUS column of ctr -n k8s.io tasks list; recover with ctr -n k8s.io tasks resume CONTAINER_ID.
# Look for PAUSED in the STATUS column
ctr -n k8s.io tasks list
# Put it back into the running state
ctr -n k8s.io tasks resume CONTAINER_ID
10. When Not to Use These Features
Checkpoint/restore from section 5 sounds attractive in description but has a narrow band of real use. CRIU restores a process's memory and file descriptor state; it does not restore the relationships that state had with the outside world. The peer of an open TCP connection knows nothing about the restore, file locks and session tokens have expired, and the database connection pool is holding sockets that are already gone. So it earns its keep on workloads with little external state — a computation whose initialization is expensive, warmed up in advance and restored on demand. Kubernetes-level container checkpoint support has changed status across releases, so check the exact field in the docs for the version you run.
The alternative runtimes in section 6 are likewise not a default. The reason to adopt Kata or gVisor is isolation, not performance, and the price paid for that isolation is startup time, memory, and compatibility. That price differs per workload, so measure it with your workload rather than someone else's benchmark. Switching the whole cluster's default runtime without measuring is the worst version of this; attaching it via RuntimeClass only to multi-tenant segments or namespaces running untrusted code is a sensible starting point. Wasm buys startup time and portability rather than isolation, and because system access is restricted, it is not a place to move existing container images unchanged.
Finally, ctr is not an operations tool. It is a debugging client for containerd developers, with no promise of a stable interface. It is excellent for reading state, but using it to create or delete containers makes kubelet's view of the world diverge from the actual one. When kubelet notices that divergence it attempts its own recovery, and there is no guarantee that recovery matches what the operator intended. On a Kubernetes node, keep the boundary: read with ctr and crictl, write with kubectl.
11. References
- containerd — Getting started (config default, installation and startup): https://github.com/containerd/containerd/blob/main/docs/getting-started.md (checked 2026-08-16)
- containerd — CRI plugin config (config version 2/3, SystemdCgroup): https://github.com/containerd/containerd/blob/main/docs/cri/config.md (checked 2026-08-16)
- containerd — ctr tasks command source (list columns, delete flags): https://github.com/containerd/containerd/tree/main/cmd/ctr/commands/tasks (checked 2026-08-16)
- containerd — ctr containers command source (list columns, info --spec): https://github.com/containerd/containerd/tree/main/cmd/ctr/commands/containers (checked 2026-08-16)
- cri-tools — crictl usage: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md (checked 2026-08-16)
12. Summary
containerd container lifecycle management revolves around the separation of Container (metadata) and Task (execution), process isolation through shims, and support for diverse runtime classes. The shim's daemonized design ensures containers survive containerd restarts, while the standardized OCI runtime spec interface integrates runtimes like runc, Kata, gVisor, and Wasm seamlessly.