containerd Networking and Storage

containerd does not implement networking and storage directly but integrates with external plugins through standard interfaces. This post analyzes network configuration via CNI, namespace management, volume mounts, device access, and security module integration.

1. CNI Integration

1.1 CNI Overview

Container Network Interface (CNI) is the standard interface for container networking. containerd calls CNI plugins to configure networks.

CNI call flow:

kubelet -> containerd (CRI RunPodSandbox)
                |
                v
        Create network namespace
                |
                v
        Call CNI plugin
        (ADD command)
                |
                v
        IP allocation, routing setup, interface creation
                |
                v
        Return result to containerd

1.2 CNI Configuration

CNI configuration file location:
  Config directory: /etc/cni/net.d/
  Binary directory: /opt/cni/bin/

containerd CNI configuration (config.toml):
  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"
    max_conf_num = 1

1.3 CNI Plugin Chain

CNI config example (10-calico.conflist):

Network configuration is defined as a plugin chain:

1. Main plugin (calico, cilium, flannel, etc.):
   - Create network interface
   - IP allocation (IPAM)
   - Routing rule setup

2. Meta plugin (bandwidth, portmap, etc.):
   - Bandwidth limiting
   - Port mapping
   - Firewall rules

Execution order:
  ADD: Main -> Meta plugins (forward)
  DEL: Meta -> Main plugins (reverse)

1.4 CNI Call Details

CNI ADD execution detail:

1. containerd determines network namespace path
   /var/run/netns/cni-abc123

2. Set CNI environment variables:
   CNI_COMMAND=ADD
   CNI_CONTAINERID=abc123
   CNI_NETNS=/var/run/netns/cni-abc123
   CNI_IFNAME=eth0
   CNI_PATH=/opt/cni/bin

3. Execute CNI plugin binary
   Pass config JSON via stdin

4. Plugin returns result via stdout:
   - Assigned IP address
   - Gateway address
   - DNS configuration
   - Routing information

5. containerd stores the result

2. Network Namespaces

2.1 Namespace Creation

Pod network namespace:

During Pod Sandbox creation:
1. Create new network namespace with unshare(CLONE_NEWNET)
2. Persist via bind mount at /var/run/netns/
3. Execute CNI plugins in this namespace
4. All containers in the Pod share this namespace

Namespace sharing:
  Pause container holds the network namespace
  App containers join the same namespace
  -> Containers in Pod can communicate via localhost

2.2 Namespace Cleanup

Namespace cleanup:

During Pod deletion:
1. CNI DEL command releases network resources
   - Return IP address
   - Delete interface
   - Remove routing rules
2. Unmount bind mount from /var/run/netns/
3. Network namespace automatically deleted

3. Volume Mounts

3.1 Mount Types

containerd manages volumes through mount configuration in the OCI spec:

Mount types:

1. bind mount:
   - Mount host file/directory into container
   - Host and container share the same data
   - Used for ConfigMap, Secret, emptyDir, etc.

2. tmpfs mount:
   - Memory-based filesystem
   - Data lost on container termination
   - Used for /dev/shm, /run, etc.

3. Special filesystems:
   - proc: /proc
   - sysfs: /sys
   - cgroup: /sys/fs/cgroup
   - devpts: /dev/pts

3.2 Mount Propagation

Mount propagation options:

1. private:
   - No mount event propagation
   - Default

2. rprivate:
   - Recursive private

3. shared:
   - Bidirectional mount event propagation
   - Mount on host -> visible in container
   - Mount in container -> visible on host

4. rshared:
   - Recursive shared

5. slave:
   - Host -> container unidirectional propagation
   - Useful for volume plugins

6. rslave:
   - Recursive slave

Kubernetes usage:
  - Controlled via MountPropagation field
  - CSI drivers typically use Bidirectional (shared)

3.3 CRI Volume Processing

Volume processing via CRI:

kubelet adds mounts to OCI spec:

1. emptyDir:
   - kubelet creates directory on host
   - Passed to container as bind mount

2. hostPath:
   - Direct bind mount of host path

3. ConfigMap/Secret:
   - kubelet creates data on tmpfs
   - Passed to container as bind mount

4. PersistentVolumeClaim:
   - kubelet mounts volume via CSI driver
   - Mounted path passed as bind mount

containerd's role:
  - Reflect kubelet-prepared mount info in OCI spec
  - runc performs the actual mount

4. Device Access

4.1 Device Mapping

Device access mechanism:

OCI spec devices section:
  linux:
    devices:
      - path: "/dev/nvidia0"
        type: "c"
        major: 195
        minor: 0
        fileMode: 438
        uid: 0
        gid: 0

Cgroup device access control:
  linux:
    resources:
      devices:
        - allow: true
          type: "c"
          major: 195
          access: "rwm"

4.2 GPU Support

GPU access (NVIDIA):

NVIDIA Container Toolkit integration:

1. nvidia-container-runtime-hook:
   - Operates as OCI runtime hook
   - Runs before container start
   - Mounts NVIDIA driver libraries into container
   - Adds GPU device nodes to container

2. CDI (Container Device Interface):
   - Device vendor-neutral standard
   - Define device specs in /etc/cdi/
   - containerd reads CDI specs and reflects in OCI spec

CDI spec example:
  cdiVersion: "0.5.0"
  kind: "nvidia.com/gpu"
  devices:
    - name: "0"
      containerEdits:
        deviceNodes:
          - path: "/dev/nvidia0"
        mounts:
          - hostPath: "/usr/lib/x86_64-linux-gnu/libnvidia-ml.so"
            containerPath: "/usr/lib/x86_64-linux-gnu/libnvidia-ml.so"

4.3 Other Devices

Other device access:

1. FPGA:
   - Expose FPGA devices via CDI specs
   - Vendor-specific device plugins

2. InfiniBand/RDMA:
   - Map /dev/infiniband/* devices
   - Share network device namespace

3. Serial/USB:
   - Direct host device mapping
   - Privileged mode or explicit device allowlist

5. SELinux Integration

5.1 SELinux Context

SELinux container security:

SELinux settings in OCI spec:
  linux:
    mountLabel: "system_u:object_r:container_file_t:s0:c1,c2"
    processLabel: "system_u:system_r:container_t:s0:c1,c2"

Components:
  - user: system_u
  - role: system_r (process) / object_r (file)
  - type: container_t (process) / container_file_t (file)
  - level: s0:c1,c2 (MCS category)

MCS (Multi-Category Security):
  - Assigns unique categories to each container
  - Prevents access to other containers' files
  - Isolation between host and container

5.2 SELinux Processing Flow

SELinux application:

1. kubelet determines Pod SELinux options
   - securityContext.seLinuxOptions
   - Automatic MCS label assignment

2. Passed to containerd via CRI
   - processLabel: process security context
   - mountLabel: file security context

3. containerd reflects in OCI spec

4. runc applies at execution:
   - Apply SELinux label to process
   - Apply SELinux label to rootfs
   - Apply SELinux label to mounts

6. AppArmor Integration

6.1 AppArmor Profiles

AppArmor container security:

Default profile: cri-containerd.apparmor.d

Key rules:
  - Filesystem access restrictions
    deny /proc/kcore r,
    deny /sys/firmware/** r,
  - Network access control
  - Capability restrictions
  - Mount operation restrictions

Profile application:
  OCI spec:
    process:
      apparmorProfile: "cri-containerd.apparmor.d"

6.2 Custom Profiles

Custom AppArmor profiles:

1. Install profile on host:
   Place profile file in /etc/apparmor.d/
   apparmor_parser -r /etc/apparmor.d/my-profile

2. Specify in Pod:
   annotations:
     container.apparmor.security.beta.kubernetes.io/app: localhost/my-profile

3. containerd reflects in OCI spec:
   process:
     apparmorProfile: "my-profile"

7. Seccomp Integration

7.1 Seccomp Profiles

Seccomp (Secure Computing):

Define allowed/blocked system calls:

Default action: SCMP_ACT_ERRNO (deny)

Allowed system calls example:
  - read, write, open, close
  - mmap, mprotect, munmap
  - socket, connect, accept
  - ...

Blocked system calls example:
  - mount, umount (prevent container escape)
  - reboot
  - kexec_load
  - ptrace (in some environments)

7.2 Seccomp Application

Seccomp profile application:

1. Kubernetes SecurityContext:
   securityContext:
     seccompProfile:
       type: RuntimeDefault

2. RuntimeDefault profile:
   - containerd/runc default Seccomp profile
   - Blocks dangerous system calls
   - Suitable for most workloads

3. Custom profile:
   securityContext:
     seccompProfile:
       type: Localhost
       localhostProfile: "profiles/my-seccomp.json"

8. Checking This on a Real Node

There are two tools and they sit at different layers. crictl calls the CRI API, so it gives you the same view kubelet has. ctr calls containerd's native API, so it shows fields that do not exist in CRI at all, such as the snapshot key or the runtime type.

8.1 Wiring up crictl

The first time you run crictl it usually stalls for a few seconds because it cannot find the socket. The cri-tools docs recommend setting the endpoints explicitly because the default connection attempt takes seconds to time out. Put them in the config file, or set the CONTAINER_RUNTIME_ENDPOINT and IMAGE_SERVICE_ENDPOINT environment variables.

# /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

8.2 Finding the Sandbox and Entering Its Network Namespace

A Pod's network is held by the sandbox, not by the app container, so always start from the sandbox.

crictl pods
crictl pods --name nginx-65899c769f-wv2gp
crictl inspectp 4dccb216c4adb

crictl pods prints six columns: POD ID, CREATED, STATE, NAME, NAMESPACE, ATTEMPT. If STATE is not Ready there is no point looking at the containers underneath. It means the sandbox never came up, and the cause is almost always CNI. If ATTEMPT is not 0, kubelet has already rebuilt the sandbox for that Pod several times, so the number itself is the signal of a repeating failure.

What crictl inspectp returns is the CRI PodSandboxStatusResponse. Under status you get id, metadata, state, created_at, network, linux, labels, annotations and runtime_handler, and network holds ip and additional_ips. If ip is empty, either CNI ADD did not succeed or its result was never stored. runtime_handler tells you which runtime handler this sandbox came up under, which is where you check that a RuntimeClass actually routed to the runtime you intended.

The other half of the response, info, is a map, and the CRI definition only states that the key may be an arbitrary string and the value should be in json format. The contents are up to the runtime, and containerd puts the runtime spec and the sandbox config there. The network namespace path arrives inside that blob too, but the key name and nesting vary by version, so check the exact field in the docs for the version you run.

Once you have the path, go inside and look at the actual interfaces.

ip netns list
ip netns exec cni-1f2e3d4c-5a6b-7c8d-9e0f-a1b2c3d4e5f6 ip addr
ip netns exec cni-1f2e3d4c-5a6b-7c8d-9e0f-a1b2c3d4e5f6 ip route

Two things matter in ip addr: does eth0 exist, and does it carry an address from the Pod CIDR range. No eth0 at all means the main CNI plugin never created the interface. An interface with no address means it died at the IPAM step. That single distinction splits plugin-side causes from IPAM-side causes. In ip route you check that the default route points at the gateway CNI handed out. If it is empty, every packet leaving the Pod dies.

If the netns file is already gone but the process is still alive, entering by PID is the only way in.

nsenter -t 12345 -n ip addr
lsns -t net

8.3 Did containerd Actually Load the CNI Config

A CNI config file existing on disk and containerd having read it are two different things. containerd's CRI Status response carries two conditions, RuntimeReady and NetworkReady. When the network is not ready the reason is NetworkPluginNotReady and the message is of the form Network plugin returns error followed by the real cause. The info map on that response carries the keys config, golang, cniconfig, lastCNILoadStatus, and a per-network lastCNILoadStatus.

crictl info

cniconfig shows the conflist containerd actually parsed and is holding. If that differs from the file on disk, either containerd never re-read the file after you edited it, or the JSON is broken and parsing failed. If lastCNILoadStatus carries an error, that string is effectively the answer.

8.4 Which Snapshot and Mounts a Container Got

For storage you go the other way and look at the app container.

crictl ps
crictl ps -a
crictl inspect 1f73f2d81bf98
ctr -n k8s.io containers list
ctr -n k8s.io containers info 1f73f2d81bf98

crictl ps prints CONTAINER ID, IMAGE, CREATED, STATE, NAME, ATTEMPT, and -a includes containers that already exited. ctr always needs the namespace. The CRI plugin uses the k8s.io containerd namespace, so leaving the flag off returns nothing and leads you to the wrong conclusion that there is nothing to inspect. ctr containers list prints three columns, CONTAINER, IMAGE and RUNTIME, and the RUNTIME column tells you which shim the container came up under, so RuntimeClass problems surface here too.

ctr containers info dumps the whole container record as JSON. The snapshotter name and the snapshot key are in there, and the spec flag narrows it to the OCI runtime spec alone. Questions like whether this container got a SELinux label, or what propagation option this mount carries, are answered by reading the mounts array and devices, mountLabel and processLabel under linux. The list of snapshotters you can actually use comes from the plugin listing.

On the host side findmnt is the fastest way to see the mounts. It prints the propagation property alongside the tree, so a mismatch between what the OCI spec says and what the kernel actually built is visible directly.

ctr plugins ls
findmnt -o TARGET,SOURCE,FSTYPE,PROPAGATION
findmnt -T /var/lib/kubelet/pods

9. config.toml Version 2 versus Version 3

Pasting a snippet you found somewhere into config.toml and getting no effect at all is common. Usually it is because the section headers change wholesale between containerd major versions. Version 2 was introduced in 1.3 and keeps all CRI settings under a single section. Version 3 arrived in 2.0 and split that one section into runtime and images. Version 2 files are still supported in 2.x and are automatically converted to version 3, but version 1 is no longer supported from 2.0 on.

# containerd 1.x — config version 2
version = 2

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/opt/cni/bin"
  conf_dir = "/etc/cni/net.d"
  max_conf_num = 1

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

# containerd 2.x — config version 3
version = 3

[plugins.'io.containerd.cri.v1.runtime'.cni]
  bin_dir = '/opt/cni/bin'
  conf_dir = '/etc/cni/net.d'
  max_conf_num = 1

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins.'io.containerd.cri.v1.images']
  snapshotter = 'overlayfs'

Do not assemble the headers by hand. Generate the default and start from that. The command prints the schema the containerd binary you actually installed understands, so there is no room to pick the wrong version.

containerd config default > /etc/containerd/config.toml
systemctl daemon-reload
systemctl restart containerd

The CNI keys are documented as follows. conf_dir is the directory in which the admin places a CNI conf, and bin_dir is the directory in which the binaries for the plugin are kept; bin_dir is marked deprecated in v2.1 and replaced by bin_dirs. max_conf_num specifies the maximum number of CNI plugin config files to load from the CNI config directory, and by default only 1 is loaded. conf_template is the file path of a golang template used to generate cni config, and if it is set containerd will generate a cni config file from the template. ip_pref specifies the strategy for selecting the main IP address for a pod, and the default selects the first ipv4 address.

The default snapshotter is overlayfs, which the docs describe as akin to Docker's overlay2 storage driver. SystemdCgroup defaults to false in both version 2 and version 3, though true is recommended on systemd-based hosts. That default matters: if you switch only kubelet to systemd and leave containerd alone, the two managers end up with different views of the cgroup hierarchy.

10. Worked Example: a Pod Stuck in ContainerCreating

A Pod will not leave ContainerCreating and the events show a CNI-related error. Follow the order and this usually takes minutes; skip the order and you will spend an hour reading config.toml first.

First, read the kubelet events. The string printed there sets the direction for everything that follows. When containerd reports that the network plugin is not ready, the reason is NetworkPluginNotReady and the message starts with Network plugin returns error. If you see that string this is not one Pod's problem, it is the whole node's CNI, so looking at the other Pods on the same node immediately tells you the blast radius.

kubectl describe pod my-app-7d9f6c8b4-xk2wp
kubectl get pods -A -o wide --field-selector spec.nodeName=node-3

Second, separate "the sandbox was never created" from "the sandbox exists but has no IP". Absent from the listing means RunPodSandbox itself failed. Present but with an empty ip under network means CNI ADD failed. The former may be a runtime problem; the latter is almost certainly a network problem.

crictl pods --name my-app-7d9f6c8b4-xk2wp
crictl inspectp 4dccb216c4adb

Third, check that containerd loaded the CNI config. Read cniconfig and lastCNILoadStatus from 8.3, and if they are empty check conf_dir on disk. If the file is clearly there and the config is still empty the JSON is broken, so parse it yourself. This is where someone leaving a stray comma while hand-editing a conflist gets caught.

crictl info
ls -l /etc/cni/net.d/
python3 -m json.tool /etc/cni/net.d/10-calico.conflist > /dev/null

Fourth, check that the plugin binaries the conflist references actually exist in bin_dir. The type value in the plugins array is the binary name that will be executed, and a single missing name stops CNI from running at all. This is the classic breakage right after a node image change or a CNI upgrade. Meta plugins like bandwidth and portmap ship in a different package from the main plugin, so they go missing quietly.

ls -l /opt/cni/bin/
grep -o '"type"[^,]*' /etc/cni/net.d/10-calico.conflist

Fifth, if everything so far is fine, suspect IPAM exhaustion. Once the range assigned to the node is fully used, CNI ADD is still invoked but fails because no address comes back. The symptom appears on one node and only when new Pods start, so it clusters by node. How to confirm it depends on the CNI implementation, so you have to look at your plugin's IPAM store.

Sixth, if it still does not resolve, look for dead namespaces. When kubelet or the node dies abnormally, CNI DEL never runs, the bind mount stays behind, and as far as IPAM is concerned that address is still in use. Comparing the namespace count against the number of Ready sandboxes exposes the gap.

ip netns list | wc -l
crictl pods

11. Failure Modes and Diagnosis Order

11.1 Leaked Network Namespaces After a kubelet Crash

The symptom is that new Pods only fail to get an IP on one node. Rebooting the node clears it, then it comes back a few days later. Diagnosis starts by comparing the namespace count with the sandbox count. A gap means sandboxes disappeared without CNI DEL running and that many IPs were never reclaimed, so the next step is finding the moment in the node journal where kubelet or containerd died without going through its cleanup path. Do not clean this up by hand. Deleting a live Pod's namespace by mistake severs its network instantly. Draining and replacing the node is faster and safer.

11.2 max_conf_num Picking the Wrong conflist

The symptom is that you replaced the CNI but Pods still attach to the old network, or a new conflist is simply not taking effect. The cause is that max_conf_num defaults to 1. Even with several files in conf_dir only one is loaded, and name order decides the winner, so an old file left behind by an installer wins if it sorts first. Diagnosis ends at checking which file the cniconfig from 8.3 came from. The fix is cleaning the directory, not raising max_conf_num.

11.3 Mount Propagation Set to private Breaking a CSI Driver

The symptom is that the CSI driver Pod looks healthy but workloads using it see an empty volume. The driver reports that it mounted, and there is nothing inside the app container. The default private from 3.2 is the cause: a mount created inside the driver container cannot propagate up to the host, and therefore cannot come back down into other containers. The field that sets this is MountPropagation, and CSI drivers typically use Bidirectional. Diagnosis is reading the propagation property on the host with findmnt, and the trap is that you have to look at the driver Pod's manifest rather than the app's.

11.4 SELinux MCS Denials That Look Like Permission Errors

The symptom is permission denied when opening a file inside the container. You check the owner and mode repeatedly and they are correct. The MCS from 5.1 is the cause: a file created by a container with a different label is denied no matter how open the mode is. It shows up most often when two Pods share a hostPath, and when a restart changes the category. The decisive clue is the host audit log. If the label is the cause the denial is recorded on the host side, and otherwise nothing appears, so that one check separates a permission problem from a label problem.

11.5 The Unexplained EPERM from seccomp RuntimeDefault

The symptom is that a particular library or runtime fails only inside the container. The error is EPERM and no permission looks missing. The default action from 7.1, SCMP_ACT_ERRNO, is the answer. A blocked call never reaches the kernel and returns an error instead, and the application reports that as a permission problem, so logs alone cannot tell you. Diagnosis is switching the profile to Unconfined briefly to see whether the symptom disappears, and if it does, finding which call was blocked and allowing exactly that in a custom profile. Unconfined is a diagnostic tool, not a fix.

12. When Not to Touch This Layer

First, most of the behaviour in this post is the CNI plugin's and kubelet's job, not containerd's. Handing out the IP, installing routes, enforcing network policy are all the plugin. The fix for a Pod that gets no IP is almost always in Calico or Cilium configuration, not containerd's.

Second, hand-editing config.toml on a managed node group gets reverted. In environments that treat nodes as replaceable, changes made directly on a node vanish at the next scale-out or upgrade. The changes that survive go through the node image, the bootstrap script, or whatever node configuration mechanism the platform offers. Editing on the node is diagnosis, not deployment.

Third, reading this layer and writing to it are different things. Reading state with crictl and ctr is safe, but deleting containers or sandboxes with these tools changes the world behind kubelet's back, and kubelet may start an unexpected recovery when it notices. If you want a container gone, delete the Pod through kubectl.

Fourth, reaching for conf_template when a standard plugin combination would do raises the debugging difficulty a level. The template exists for generating per-node configuration, and without that requirement a single static conflist is better, because when something breaks you open the file and read it.

13. References

CRI plugin config: https://github.com/containerd/containerd/blob/main/docs/cri/config.md (checked 2026-08-16)
Getting Started: https://github.com/containerd/containerd/blob/main/docs/getting-started.md (checked 2026-08-16)
Snapshotters: https://github.com/containerd/containerd/blob/main/docs/snapshotters/README.md (checked 2026-08-16)
crictl docs: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md (checked 2026-08-16)
Debugging nodes with crictl: https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/ (checked 2026-08-16)
Container runtimes: https://kubernetes.io/docs/setup/production-environment/container-runtimes/ (checked 2026-08-16)
CRI API definition: https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1/api.proto (checked 2026-08-16)

14. Summary

containerd networking and storage follows a delegation model through standard interfaces. Network configuration via CNI, mount management via OCI spec, device access via CDI, and security isolation via SELinux/AppArmor/Seccomp are the key pillars. This standards-based design allows containerd to flexibly integrate with various networking solutions and security modules.

[containerd] Networking and Storage