LabHub

Blog

Mastering the NVIDIA GPU Operator — From Install and Deployment to MIG Partitioning

2026-07-07 · 8 min read · #kubernetes#gpu#nvidia#mig#devops#mlops

한국어English日本語中文

Introduction — Escaping GPU-Node Setup Hell

Anyone who has manually attached a GPU node to a Kubernetes cluster knows the drill: install the NVIDIA driver matched to the kernel version, install the container toolkit, adjust the runtime config, deploy the device plugin, attach a monitoring exporter — and if the version compatibility of even one of these five layers slips, pods cannot see the GPU. With ten nodes you do this ten times, and when a kernel update lands you start over.

The NVIDIA GPU Operator automates this whole stack with the Kubernetes operator pattern. The administrator installs one Helm chart, declares the desired state, and the operator deploys the needed components to each node as containers and keeps reconciling them. This article covers everything from installation and verification to the killer feature of A100/H100-class GPUs: MIG (Multi-Instance GPU) partitioning — with real commands. If you want to warm up on Kubernetes basics first, try this site's Kubernetes playground.

What the GPU Operator Deploys

Installing the operator lays down the following operands as DaemonSets in the gpu-operator namespace. Each corresponds to one job you used to do by hand.

Installation — One Helm Chart

Three prerequisites: nodes with NVIDIA GPUs, a supported container runtime (containerd etc.), and Node Feature Discovery (installed with the operator by default).

# 1) Add the NVIDIA Helm repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# 2) Install (check the docs for the latest version — v26.3.3 at time of writing)
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3

Two common variants depending on your environment:

# If the driver is already installed on the node (DGX OS, manual install, etc.)
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3 \
  --set driver.enabled=false

# If the container toolkit is also preinstalled, add:
#   --set toolkit.enabled=false

Check installation status:

kubectl get pods -n gpu-operator
# You are healthy when driver-daemonset, container-toolkit, device-plugin,
# dcgm-exporter, gpu-feature-discovery, operator-validator are Running/Completed

Verification — Can a Pod See the GPU?

The most reliable check is a test pod that requests a GPU.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  containers:
    - name: nvidia-smi
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ['nvidia-smi']
      resources:
        limits:
          nvidia.com/gpu: 1

kubectl apply -f gpu-smoke-test.yaml
kubectl logs gpu-smoke-test
# Success when nvidia-smi prints your GPU model

Also worth checking the node labels GFD attached — useful for writing node selectors:

kubectl get node <node-name> -o jsonpath='{.metadata.labels}' | jq . | grep nvidia
# nvidia.com/gpu.product, nvidia.com/gpu.memory, nvidia.com/gpu.count, etc.

MIG — Slicing One GPU at the Hardware Level

Datacenter GPUs like the A100/H100 are too big as a single unit. When one small inference workload occupies an entire 80 GB GPU, the rest of the capacity idles. MIG (Multi-Instance GPU), supported since the Ampere generation, splits one GPU into up to seven hardware-isolated instances.

The key phrase is "hardware-isolated." Each MIG instance is physically allocated its own share of SMs (compute units), L2 cache, and memory bandwidth. That is the decisive difference from time-slicing — with time-slicing, pods take turns on the GPU, so one pod's burst becomes another pod's latency; with MIG, your performance holds no matter what the neighboring instance is doing. This is why MIG is preferred for multi-tenant clusters, inference serving, and per-team GPU allocation.

Profile names follow the format <GI count>g.<memory>gb. One H100 80GB, for example, can typically be split like this:

Profile      Max instances    Rough use case
1g.10gb   x7                 7 small inference services
2g.20gb   x3                 3 medium inference/experiments
3g.40gb   x2                 2 mid-to-large training runs
7g.80gb   x1                 the whole GPU, unpartitioned
(Mixes are possible too: e.g. one 3g.40gb + one 2g.20gb + two 1g.10gb)

On an A100 40GB the same principle yields 1g.5gbx7, 2g.10gbx3, 3g.20gbx2, and so on. Exact combination rules differ per GPU model — treat the profile tables in the NVIDIA MIG User Guide as the source of truth.

MIG Strategy — single vs. mixed

To use MIG with the GPU Operator, you first choose a strategy at install time. This choice changes how resources are advertised to Kubernetes.

# Install with the single strategy
helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.3.3 \
  --set mig.strategy=single

# In cloud environments (GKE/EKS etc.) that require a reboot, add:
#   --set migManager.env[0].name=WITH_REBOOT \
#   --set-string migManager.env[0].value=true

To change only the strategy on an existing cluster, patch the ClusterPolicy:

kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type='json' \
  -p='[{"op":"replace", "path":"/spec/mig/strategy", "value":"mixed"}]'

Applying MIG — With a Single Node Label

MIG Manager's mechanism is elegantly simple: it watches the nvidia.com/mig.config node label, and when the label changes, it applies that profile to the hardware.

# Split an H100 node into seven 1g.10gb instances
kubectl label nodes <node-name> nvidia.com/mig.config=all-1g.10gb --overwrite

Built-in profiles include all-disabled (turn MIG off), all-1g.10gb, all-2g.20gb, all-3g.40gb, and all-balanced (a balanced mix of sizes). Track the application via labels:

kubectl get node <node-name> -o jsonpath='{.metadata.labels}' | jq . | grep mig
# nvidia.com/mig.config.state transitions
#   pending -> (rebooting, if needed) -> success

After success, check resources to see the result:

kubectl describe node <node-name> | grep -A5 'Allocatable'
# single strategy: nvidia.com/gpu: 7  (gpu.product label shows MIG-1g.10gb)
# mixed  strategy: per-profile resources like nvidia.com/mig-1g.10gb: 7

Under the mixed strategy, pods request a specific profile like this:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

Custom Profiles — Mixing Sizes Within One Card

When the built-ins are not enough (say, "GPU 0 gets a 3g+1g+1g mix and the rest keep MIG off"), define your own via a ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-disabled:
        - devices: all
          mig-enabled: false
      inference-mix:
        - devices: [0]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 5
            "2g.20gb": 1
        - devices: [1, 2, 3]
          mig-enabled: false

# Point MIG Manager at this ConfigMap via a ClusterPolicy patch
kubectl patch clusterpolicies.nvidia.com/cluster-policy \
  --type='json' \
  -p='[{"op":"replace", "path":"/spec/migManager/config/name", "value":"custom-mig-config"}]'

# Apply the custom profile to a node
kubectl label nodes <node-name> nvidia.com/mig.config=inference-mix --overwrite

Note: since v26.3.0, MIG Manager queries the node's hardware via NVML at startup and generates the list of possible profiles at runtime, writing a per-node ConfigMap. When a new GPU model ships, you no longer wait for an updated built-in profile list.

Operational Cautions — Non-Incidents If You Know Them in Advance

Closing

The GPU Operator turned the repetitive labor of "GPU node setup" into one declarative resource, and MIG Manager lets you handle the utilization problem of very expensive GPUs with a single label line on top of it. In summary: install with Helm, choose a strategy (single/mixed), partition with the nvidia.com/mig.config label, confirm with mig.config.state, observe with DCGM — those five sentences are the whole story. What remains is finding the profile that fits your workloads, and your monitoring data will tell you that.

To shore up Linux and Kubernetes fundamentals alongside, try the Linux terminal and the Kubestronaut quiz.

References

Read the original: https://www.youngju.dev/blog/2026-07-07-nvidia-gpu-operator-mig-guide.en

Comments

No comments yet.

Sign in to leave a comment