LabHub

Blog

Kubernetes Dynamic Resource Allocation and GPU Scheduling

2026-03-06 · 20 min read · #kubernetes#dra#gpu#dynamic-resource-allocation#scheduling#2026-03

한국어English日本語

Kubernetes DRA GPU Scheduling

Introduction

Scheduling hardware accelerators such as GPUs on Kubernetes has been a hard problem for operators for a long time. Because the existing Extended Resource model allocated GPUs as a simple integer count, like nvidia.com/gpu: 1, scheduling on a GPU's detailed attributes (VRAM capacity, MIG partition, a specific model) was fundamentally impossible.

DRA (Dynamic Resource Allocation) - introduced as alpha in Kubernetes 1.26 to solve this problem, substantially redesigned in 1.31 and promoted to beta in 1.32 - is a framework for requesting and allocating complex hardware resources such as GPUs, FPGAs and network devices in a structured way.

The Limits of the Existing Extended Resource Model

In the existing model, GPU allocation looked like this:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-legacy
spec:
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:24.01-py3
      resources:
        limits:
          nvidia.com/gpu: 2

This approach has a few fundamental limitations:

DRA removes all of these limitations and fundamentally improves the scheduling quality of GPU workloads. In this article, we cover DRA systematically, from its architecture through practical deployment to operational troubleshooting.

DRA Architecture Overview

DRA belongs to Kubernetes' resource.k8s.io/v1beta1 API group (as of 1.32) and consists of the following four core resources.

The Core Resource Structure

DeviceClass: defines a type of device. Create a DeviceClass per GPU model so that a workload can request a specific GPU type. It plays a role similar to StorageClass for storage.

ResourceClaim: represents an actual device allocation request. Just as a PersistentVolumeClaim requests storage, a ResourceClaim requests a device from a particular DeviceClass. It can exist independently of a Pod's lifecycle, so a device can be shared between several Pods, or kept the same across a Pod restart.

ResourceClaimTemplate: a template for creating a ResourceClaim automatically when a Pod is created. Use it when each Pod needs its own dedicated ResourceClaim.

ResourceSlice: the list of devices that the DRA driver installed on a node reports to the API server. The scheduler consults ResourceSlices to learn which devices are available on which node.

The Scheduling Flow

DRA-based scheduling proceeds through the following steps:

  1. Pod submission: the user creates a Pod that references a ResourceClaim.
  2. Scheduler filtering: kube-scheduler uses the ResourceSlice information to filter for nodes where a device of the requested DeviceClass is available.
  3. Structured parameter matching: the scheduler matches the ResourceClaim's requirements directly against the available devices on each node. This is the heart of DRA's "Structured Parameters" model.
  4. Allocation decision: the scheduler allocates a specific device on a specific node to the ResourceClaim.
  5. Device preparation: that node's DRA driver (the kubelet plugin) prepares the allocated device so the container can use it (CDI device configuration and so on).
  6. Container start: kubelet starts the container with the prepared device.

Unlike the existing Device Plugin approach, in DRA the scheduler itself understands the device attributes and decides the optimal placement. That is a fundamentally different approach from the Device Plugin merely reporting an "available count".

Installing and Configuring the DRA Driver

Prerequisites

Using DRA requires Kubernetes 1.32 or later, with the DynamicResourceAllocation feature gate enabled. In Kubernetes 1.32 it is beta, so it is enabled by default.

# check the cluster version
kubectl version --short

# check that the DRA feature gate is enabled (needed on kube-apiserver, kube-scheduler and kubelet)
# on a kubeadm-based cluster this is set in the ClusterConfiguration
kubectl get configmap -n kube-system kubeadm-config -o yaml | grep -A 5 "featureGates"

# check the DRA-related API resources
kubectl api-resources | grep resource.k8s.io
# example output:
# deviceclasses              resource.k8s.io/v1beta1   false   DeviceClass
# resourceclaims             resource.k8s.io/v1beta1   true    ResourceClaim
# resourceclaimtemplates     resource.k8s.io/v1beta1   true    ResourceClaimTemplate
# resourceslices             resource.k8s.io/v1beta1   false   ResourceSlice

Installing the NVIDIA DRA Driver

NVIDIA provides an official DRA driver, nvidia-dra-driver. It is a separate project from the existing k8s-device-plugin, and it fully supports DRA's structured parameter model.

# add the NVIDIA DRA driver Helm chart repository
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# install nvidia-dra-driver
# if the NVIDIA GPU Operator is already installed, the device-plugin has to be disabled
helm install nvidia-dra-driver nvidia/nvidia-dra-driver \
  --namespace nvidia-dra-driver \
  --create-namespace \
  --set controller.enabled=true \
  --set kubeletPlugin.enabled=true \
  --version v0.8.0

# confirm the installation
kubectl get pods -n nvidia-dra-driver
# NAME                                              READY   STATUS    RESTARTS   AGE
# nvidia-dra-driver-controller-7d8f9b6c4d-k2xvn    1/1     Running   0          2m
# nvidia-dra-driver-kubelet-plugin-node1-abcde      1/1     Running   0          2m
# nvidia-dra-driver-kubelet-plugin-node2-fghij      1/1     Running   0          2m

# confirm the ResourceSlices were created - the driver reports the node's GPU information
kubectl get resourceslices -o wide
# NAME                              DRIVER               NODE    POOL        DEVICES
# gpu-node1-nvidia-gpu-slice-0      gpu.nvidia.com       node1   nvidia-gpu  4
# gpu-node2-nvidia-gpu-slice-0      gpu.nvidia.com       node2   nvidia-gpu  8

Inspecting a ResourceSlice in Detail

Once the driver is installed correctly, each GPU node's device information is reported as a ResourceSlice. That is how the scheduler learns each GPU's detailed attributes.

# view a specific node's ResourceSlice in detail
kubectl get resourceslice gpu-node1-nvidia-gpu-slice-0 -o yaml

The example output shows that a rich set of attributes is reported for each GPU: model name, VRAM, UUID, architecture, MIG support and more. Those attributes are exactly what DeviceClass and ResourceClaim use as filter conditions.

Defining DeviceClasses and Managing GPU Tiers

Defining a DeviceClass per GPU Model

Like StorageClass, DeviceClass defines a "class" of device. Once the cluster administrator defines them up front, workload developers can request the GPU they want simply by referencing the DeviceClass name.

# gpu-deviceclasses.yaml
apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com-a100
spec:
  selectors:
    - cel:
        expression: "device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].productName == 'NVIDIA A100 80GB PCIe'"
  config:
    - opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: gpu.nvidia.com/v1alpha1
          kind: GpuConfig
          sharing:
            strategy: TimeSlicing
            timeSlicingConfig:
              interval: Long
---
apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com-h100
spec:
  selectors:
    - cel:
        expression: "device.driver == 'gpu.nvidia.com' && device.attributes['gpu.nvidia.com'].productName == 'NVIDIA H100 80GB HBM3'"
  config:
    - opaque:
        driver: gpu.nvidia.com
        parameters:
          apiVersion: gpu.nvidia.com/v1alpha1
          kind: GpuConfig
          sharing:
            strategy: TimeSlicing
            timeSlicingConfig:
              interval: Short
---
apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: gpu.nvidia.com-a100-mig-3g20gb
spec:
  selectors:
    - cel:
        expression: >-
          device.driver == 'gpu.nvidia.com' &&
          device.attributes['gpu.nvidia.com'].productName == 'NVIDIA A100 80GB PCIe' &&
          device.attributes['gpu.nvidia.com'].migProfile == '3g.20gb'

The example above defines three DeviceClasses:

MIG Partitioning Strategy

Using MIG on an NVIDIA A100/H100 splits one physical GPU into several independent instances, which maximizes utilization. In DRA you can define a DeviceClass per MIG profile, which makes fine-grained GPU resource management possible.

MIG profileCompute slicesVRAMMax instances (on A100)Primary use
1g.5gb1/75GB7Inference (small models), dev/test
1g.10gb1/710GB7Inference (medium models)
2g.10gb2/710GB3Small-scale training, batch inference
3g.20gb3/720GB2Mid-scale training, fine-tuning
4g.40gb4/740GB1Large-scale training
7g.80gb7/780GB1The whole GPU (same as no MIG)

A100 vs H100 Specifications

When designing a DeviceClass it matters that you understand the characteristics of the GPU model precisely.

CategoryNVIDIA A100 80GBNVIDIA H100 80GB
ArchitectureAmpereHopper
FP16 performance312 TFLOPS989 TFLOPS
FP8 performanceNot supported1,979 TFLOPS
VRAM80GB HBM2e80GB HBM3
Memory bandwidth2 TB/s3.35 TB/s
NVLink bandwidth600 GB/s900 GB/s
Max MIG instances77
TDP300W700W
DRA driver supportnvidia-dra-driver v0.6+nvidia-dra-driver v0.7+

Because FP8 arithmetic and the Transformer Engine give the H100 more than 3x the LLM training/inference performance of the A100, it is worth establishing a policy that allocates the H100 DeviceClass to LLM workloads first.

Pod Scheduling with ResourceClaim

Basic ResourceClaim Usage

This approach creates a ResourceClaim directly and references it from the Pod. Use it when the device has to be managed independently of the Pod lifecycle, or shared between several Pods.

# create a standalone ResourceClaim
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaim
metadata:
  name: training-gpu-claim
  namespace: ml-workloads
spec:
  devices:
    requests:
      - name: gpu
        deviceClassName: gpu.nvidia.com-a100
        count: 4
    constraints:
      - requests: ['gpu']
        matchAttribute: 'gpu.nvidia.com/nvlinkInterconnect'
---
# a Pod that references the ResourceClaim
apiVersion: v1
kind: Pod
metadata:
  name: distributed-training
  namespace: ml-workloads
spec:
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:24.01-py3
      command: ['torchrun', '--nproc_per_node=4', 'train.py']
      resources:
        claims:
          - name: training-gpus
  resourceClaims:
    - name: training-gpus
      resourceClaimName: training-gpu-claim
  restartPolicy: Never

The thing to notice in the example above is the constraints field. matchAttribute sets a constraint so that the 4 A100 GPUs are allocated as a set interconnected by NVLink. That has a decisive effect on inter-GPU communication performance during distributed training.

Using a ResourceClaimTemplate

For a workload that creates several Pods, such as a Job or a Deployment, use a ResourceClaimTemplate so that a dedicated ResourceClaim is created automatically for each Pod.

# deploying a MIG-based inference service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference-service
  namespace: ml-workloads
spec:
  replicas: 6
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      containers:
        - name: inference-server
          image: nvcr.io/nvidia/tritonserver:24.01-py3
          ports:
            - containerPort: 8000
              name: http
            - containerPort: 8001
              name: grpc
          resources:
            claims:
              - name: inference-gpu
      resourceClaims:
        - name: inference-gpu
          resourceClaimTemplateName: mig-inference-template
---
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
  name: mig-inference-template
  namespace: ml-workloads
spec:
  spec:
    devices:
      requests:
        - name: mig-gpu
          deviceClassName: gpu.nvidia.com-a100-mig-3g20gb
          count: 1

In this configuration 6 Triton Inference Server replicas each get 1 MIG 3g.20gb slice of an A100. Running 6 inference instances on 2 physical A100s improves resource utilization considerably compared with the existing Extended Resource model, which would have had to allocate 2 whole GPUs.

Inline Device Requests

In simple cases you can request a device directly from the Pod spec, without creating a separate ResourceClaim or ResourceClaimTemplate.

apiVersion: v1
kind: Pod
metadata:
  name: quick-gpu-job
  namespace: ml-workloads
spec:
  containers:
    - name: compute
      image: nvcr.io/nvidia/cuda:12.3.1-runtime-ubuntu22.04
      command: ['python3', 'benchmark.py']
      resources:
        claims:
          - name: gpu-req
  resourceClaims:
    - name: gpu-req
      resourceClaimTemplateName: ''
      source:
        devices:
          requests:
            - name: gpu
              deviceClassName: gpu.nvidia.com-h100
              count: 1
  restartPolicy: Never

Multi-cloud DRA Comparison (AKS vs GKE vs EKS)

Here is a comparison of DRA support and GPU instance options across the cloud providers.

CategoryAKS (Azure)GKE (Google)EKS (AWS)
Minimum Kubernetes version1.31+ (Preview)1.32+ (Preview)1.32+ (Preview)
DRA feature gateMust be enabled manuallyAuto-enabled per GKE channelManaged via an EKS addon
GPU instance (A100)NC A100 v4a2-highgpu / a3-highgpup4d.24xlarge
GPU instance (H100)ND H100 v5a3-ultragpu-8gp5.48xlarge
NVIDIA DRA driverManual Helm installIntegrated into the GKE GPU OperatorEKS NVIDIA addon
MIG supportSupported (manual setup)Supported (GKE MIG manager)Supported (manual setup)
Node autoscaling integrationKarpenter / Cluster AutoscalerNAP / KarpenterKarpenter
Price (A100 80GB, per hour)~$3.67~$3.67~$32.77 (all 8 GPUs)
Main limitationsPreview feature, no SLARapid channel onlyOnly certain regions

Configuration Differences by Cloud

On GKE you can enable DRA when creating the GPU node pool:

# create a DRA-capable GPU node pool on GKE
gcloud container node-pools create gpu-pool-h100 \
  --cluster=ml-cluster \
  --zone=us-central1-c \
  --machine-type=a3-highgpu-8g \
  --accelerator=type=nvidia-h100-80gb,count=8 \
  --num-nodes=2 \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=4 \
  --node-labels="gpu-type=h100" \
  --metadata="install-nvidia-driver=True"

# enable the DRA feature gate (GKE Rapid channel)
gcloud container clusters update ml-cluster \
  --zone=us-central1-c \
  --release-channel=rapid \
  --enable-kubernetes-unstable-apis=resource.k8s.io/v1beta1

On EKS the DRA driver is managed through an EKS managed addon:

# install the NVIDIA DRA driver addon on the EKS cluster
aws eks create-addon \
  --cluster-name ml-cluster \
  --addon-name nvidia-dra-driver \
  --addon-version v0.8.0-eksbuild.1 \
  --region us-east-1

# create the GPU node group (Karpenter NodePool)
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-h100-pool
spec:
  template:
    spec:
      requirements:
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values: ["p5.48xlarge"]
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["on-demand"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-nodes
  limits:
    cpu: "1000"
    memory: 4000Gi
    nvidia.com/gpu: "64"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30m
EOF

The maturity of DRA and the depth of the integration differ from provider to provider, so before deploying to production you must read that provider's DRA support documentation and, where possible, review a separate support agreement covering the preview/beta feature.

Migrating from the Extended Resource Model to DRA

Migrating from Extended Resource-based GPU allocation to DRA affects the whole cluster, so it has to be done in stages.

The Migration Stages

Stage 1 - Assessment and inventory (1-2 weeks)

Work out how GPUs are currently being used in the cluster.

# list every Pod currently using a GPU through the Extended Resource model
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] |
    select(.spec.containers[].resources.limits["nvidia.com/gpu"] != null) |
    [.metadata.namespace, .metadata.name,
     (.spec.containers[].resources.limits["nvidia.com/gpu"] // "0")] |
    @tsv' | \
  column -t -s $'\t'

# GPU allocation per node
kubectl get nodes -l nvidia.com/gpu.present=true -o json | \
  jq -r '.items[] |
    [.metadata.name,
     .status.capacity["nvidia.com/gpu"],
     .status.allocatable["nvidia.com/gpu"]] |
    @tsv' | \
  column -t -s $'\t' -N "NODE,CAPACITY,ALLOCATABLE"

Stage 2 - Preparing the DRA infrastructure (1 week)

  1. Upgrade Kubernetes to 1.32 or later.
  2. Confirm the DRA feature gate is enabled.
  3. Install the NVIDIA DRA driver.
  4. Do not remove the existing NVIDIA Device Plugin yet (they can coexist).

Stage 3 - Defining the DeviceClasses (1-2 days)

Create a DeviceClass for every GPU model present in the cluster. Use the YAML examples above as a reference and define the DeviceClasses to match your production GPU inventory.

Stage 4 - Converting the pilot workload (2-4 weeks)

Convert non-critical workloads to DRA first. Validate in development/staging, then extend to low-priority production workloads.

Stage 5 - Full conversion (2-4 weeks)

Convert every GPU workload to DRA and remove the existing NVIDIA Device Plugin. At this stage you must have a rollback plan prepared.

Cautions During the Migration

Operational Cautions and Monitoring

Capacity Planning

GPU capacity planning becomes more fine-grained in a DRA environment. When using MIG you have to consider not only the number of physical GPUs but also the mix of MIG profiles.

Recommended capacity planning principles:

Monitoring Setup

In a DRA environment you should monitor the following metrics:

# monitor ResourceClaim status
kubectl get resourceclaims --all-namespaces -o custom-columns=\
  'NAMESPACE:.metadata.namespace,'\
  'NAME:.metadata.name,'\
  'DEVICECLASS:.spec.devices.requests[0].deviceClassName,'\
  'ALLOCATED:.status.allocation.devices.results[0].device'

# check for ResourceClaims stuck in Pending (detecting allocation failures)
kubectl get resourceclaims --all-namespaces \
  --field-selector="status.allocation=" \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGE:.metadata.creationTimestamp'

# check utilization per DeviceClass
kubectl get resourceslices -o json | \
  jq -r '[.items[].spec.devices[] |
    {driver: .basic.attributes["gpu.nvidia.com"].productName.stringValue}] |
    group_by(.driver) |
    map({gpu_model: .[0].driver, total: length}) |
    .[] | [.gpu_model, .total] | @tsv'

Collecting Prometheus Metrics

The NVIDIA DRA driver exposes Prometheus metrics. Include the following key metrics in your monitoring dashboard:

Metric nameDescriptionAlert threshold
dra_resource_claims_totalTotal number of ResourceClaims-
dra_resource_claims_pendingNumber of pending ResourceClaimsWarn when pending 5 minutes or more
dra_resource_claims_allocatedNumber of allocated ResourceClaims-
dra_devices_availableAvailable devices per DeviceClassWarn when availability drops under 10%
dra_devices_allocatedNumber of allocated devices-
dra_allocation_duration_secondsTime taken to allocateWarn when p99 exceeds 30 seconds
dra_plugin_errors_totalNumber of driver errorsCritical at 5 or more within 1 minute

Failure Cases and Recovery Procedures

Case 1: A ResourceClaim Stays Pending Forever

Symptom: the Pod never leaves the Pending state, and kubectl describe pod shows the message "waiting for ResourceClaim to be allocated".

Cause analysis:

Recovery procedure:

# 1. check the ResourceClaim status
kubectl describe resourceclaim training-gpu-claim -n ml-workloads

# 2. check whether a device matching the requested DeviceClass exists
kubectl get resourceslices -o json | \
  jq '.items[].spec.devices[] |
    select(.basic.attributes["gpu.nvidia.com"].productName.stringValue == "NVIDIA A100 80GB PCIe")'

# 3. check the DRA driver status
kubectl get pods -n nvidia-dra-driver
kubectl logs -n nvidia-dra-driver -l app=nvidia-dra-driver-kubelet-plugin --tail=50

# 4. look for DRA-related errors in the scheduler logs
kubectl logs -n kube-system -l component=kube-scheduler --tail=100 | grep -i "dra\|resourceclaim\|deviceclass"

# 5. if necessary, delete the ResourceClaim and recreate it
kubectl delete resourceclaim training-gpu-claim -n ml-workloads
kubectl apply -f training-gpu-claim.yaml

Case 2: A GPU Device Leak After a Node Failure

Symptom: the node went NotReady and then recovered, but its GPUs still show as "allocated" in the ResourceSlice and are not handed to new Pods.

Cause: the node failure killed the DRA driver's kubelet plugin abnormally, so the device release procedure never completed.

Recovery procedure:

  1. Check whether the ResourceClaim of the Pod that was running on that node still exists.
  2. If the Pod was deleted, the ResourceClaim should have gone with it (when there is an owner reference); if it did not, delete it by hand.
  3. Restart the DRA driver's kubelet plugin.
# restart the DRA driver plugin Pod on that node
kubectl delete pod -n nvidia-dra-driver -l app=nvidia-dra-driver-kubelet-plugin \
  --field-selector spec.nodeName=problematic-node

# confirm the ResourceSlice updated correctly (wait 30 seconds after the restart)
sleep 30
kubectl get resourceslices --field-selector spec.nodeName=problematic-node -o yaml

Case 3: Existing Workloads Interrupted During a DRA Driver Upgrade

Symptom: while upgrading the DRA driver with Helm, the kubelet plugin DaemonSet's Pods restart and the GPU workloads running on those nodes throw device access errors.

Preventive measures:

Case 4: DeviceClass Matching Fails Because of a CEL Expression Error

Symptom: the DeviceClass was created but the ResourceClaim is never allocated. kubectl describe reports that no device matches.

Cause: the attribute name in the DeviceClass selectors.cel.expression is wrong, or the comparison value differs from what the device actually reports.

How to debug it:

# check the attribute names and values the device actually reports
kubectl get resourceslices -o json | \
  jq '.items[0].spec.devices[0].basic.attributes'

# example output:
# {
#   "gpu.nvidia.com": {
#     "driverVersion": {"stringValue": "550.54.15"},
#     "productName": {"stringValue": "NVIDIA A100 80GB PCIe"},
#     "architecture": {"stringValue": "Ampere"},
#     "cudaComputeCapability": {"versionValue": "8.0"},
#     "memory": {"quantityValue": "80Gi"}
#   }
# }
# confirm the attribute names used in the DeviceClass CEL expression match the output above exactly.

Case 5: The ResourceClaim Quota Is Exceeded

Symptom: the namespace's ResourceQuota rejects the ResourceClaim creation.

In a DRA environment you can set a per-namespace limit on the number of ResourceClaims. Be careful in particular with a Deployment that uses a ResourceClaimTemplate and has many replicas, since it can hit the quota faster than you expect.

Production Deployment Checklist

Work through the following checklist before deploying DRA-based GPU workloads to production.

Infrastructure preparation

DeviceClass design

Workload configuration

RBAC and security

Monitoring and alerting

Incident response

Performance validation

References

Read the original: https://www.youngju.dev/blog/kubernetes/2026-03-06-kubernetes-dynamic-resource-allocation-gpu-scheduling.en

Comments

No comments yet.

Sign in to leave a comment