Scheduling

Dynamic Resource Allocation

1.Why DRA ?

  • Device plugin : only says how many, limits: nvidia.com/gpu: 2. No overcommit, no sharing between containers, no pick by attribute, config per node.
  • DRA : says which. The scheduler picks specific devices from the driver's ResourceSlices by CEL on attributes and capacity. The kubelet has the driver prepare them through CDI.
  • Status : GA since v1.34, gate locked on since v1.35. v1.37 serves only resource.k8s.io/v1 by default.
AspectDevice pluginDRA
Asks withinteger in limitsa ResourceClaim
Picks devicekubeletscheduler, by CEL
Inventorynode allocatableResourceSlices
Sharingnonecontainers, Pods
Configper nodeper claim
Node setupAllocate RPCPrepare + CDI
Partitionsstaticpartitionable (beta)
Healthdrops from countdevice taints (GA)

Both can run. One resource name on one node comes from the plugin or DRA, not both.

2.The object model

Like PVC and StorageClass (Storage) : the claim asks, the class categorizes, the slice is the inventory.

  • DeviceClass (cluster) : CEL selectors, optional config. Made by an admin or a driver chart.
  • ResourceSlice (cluster) : devices per pool. The driver owns it and overwrites manual edits.
  • ResourceClaim : the request. Status has allocation, reservedFor.
  • ResourceClaimTemplate : one claim per Pod, deleted with the Pod (Owners and GC).
  • DeviceTaintRule (cluster) : taints devices by driver, pool, device. In v1 since v1.37.
DRA driver Cluster admin publishes creates ResourceSlice cluster, per pool DeviceClass cluster, CEL selectors matches ResourceClaimTemplate namespaced ResourceClaim namespaced deviceClassName per Pod Pod spec.resourceClaims[] each entry : exactly one of the two resourceClaimTemplateName resourceClaimName Container resources.claims[] name (+ request)

3.One GPU, two containers

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: big-gpu }
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.example.com
          selectors:
          - cel:
              expression: device.capacity["gpu.example.com"]
                .memory.compareTo(quantity("40Gi")) >= 0
---
apiVersion: v1
kind: Pod
metadata: { name: trainer }
spec:
  resourceClaims:
  - name: gpu
    resourceClaimTemplateName: big-gpu
  containers:
  - name: main
    image: ubuntu:24.04
    command: [sleep, infinity]
    resources: { claims: [ { name: gpu } ] }
  - name: sidecar       # same GPU
    image: ubuntu:24.04
    command: [sleep, infinity]
    resources: { claims: [ { name: gpu } ] }

4.Wiring rules

  • Each spec.resourceClaims[] entry has a name and exactly one of resourceClaimName, resourceClaimTemplateName.
  • A named claim must already exist in the Pod's namespace, or the Pod stays Pending.
  • The Pod list only makes it available. A container gets the device only if its resources.claims names it.

5.Inside a request

  • devices.requests[] : a name plus exactly, or firstAvailable[] (ordered fallback subrequests).
  • exactly : deviceClassName, selectors, allocationMode ExactCount (default, count 1) or All, adminAccess, tolerations, capacity.
  • constraints[] : matchAttribute, distinctAttribute. config[].opaque : driver + parameters.
What CEL sees

device.driver, device.attributes["dom"].x, and device.capacity["dom"].x, a Quantity (use compareTo).

A missing key is an error that aborts allocation, not a skip. Guard with has(), or optional chaining with orValue().

In short : A Pod asks through ResourceClaims that name a DeviceClass. The scheduler picks real devices from ResourceSlices with CEL, and the driver prepares them on the node.

1.From claim to device

  1. The kubelet plugin publishes ResourceSlices.
  2. resourceclaim-controller creates claims from templates.
  3. DynamicResources runs CEL, picks devices first-fit.
  4. It writes allocation + reservedFor, then binds the Pod.
  5. The kubelet calls NodePrepareResources (gRPC).
  6. The driver returns CDI device IDs.
  7. The runtime injects them into the containers.
  8. NodeUnprepareResources after the Pod stops.
  9. Last Pod gone : deallocated, template claim deleted.

The rest of the cycle : Advanced Scheduling.

resourceclaim-controller in kube-controller-manager kube-scheduler DynamicResources plugin 2. claim from template 9. clean up 3. read, CEL, pick 4. allocate + bind kube-apiserver 1. ResourceSlices Pod bound DRA driver kubelet plugin, per node kubelet 5. Prepare 6. CDI IDs 8. Unprepare 7. start with CDI container runtime devices injected The plugin registers with the kubelet in /var/lib/kubelet/plugins_registry

2.Sharing a device

  • Template : each Pod gets its own claim and devices.
  • Shared claim : many Pods name one ResourceClaim, 256 at most (reservedFor).
  • Containers : one Pod claim listed in several containers.
  • How they share at runtime (time-slicing, MPS) is up to the driver and its opaque config.
Template own device each Pod A Pod B gpu-tpl claim A claim B gpu-0 gpu-1 Shared claim many Pods Pod C Pod D shared-gpu reservedFor C, D max 256 gpu-2 One Pod two containers ctr0 ctr1 Pod E claim gpu gpu-3

3.Features in v1.37

FeatureKey fieldsFeature gateMaturity
Core DRAv1 kinds, pod spec.resourceClaimsDynamicResourceAllocationGA 1.34, locked 1.35
Prioritized listrequests[].firstAvailable[]DRAPrioritizedListGA 1.36, locked 1.37
Admin accessexactly.adminAccess + namespace labelDRAAdminAccessGA 1.36
Device taintsdevices[].taints[], exactly.tolerations[]DRADeviceTaintsGA 1.37
DeviceTaintRuledeviceSelector {driver, pool, device}DRADeviceTaintRulesGA 1.37
Extended resourceextendedResourceName : limits served by DRADRAExtendedResourceGA 1.37
Device statusclaim status.devices[]DRAResourceClaimDeviceStatusGA 1.37
Partitionable devicessharedCounters[], consumesCounters[]DRAPartitionableDevicesBeta 1.36, on
Consumable capacityallowMultipleAllocations, capacity.requestsDRAConsumableCapacityBeta 1.36, on
Binding conditionsbindingConditions, bindsToNodeDRADeviceBindingConditionsBeta 1.36, on
Resource healthallocatedResourcesStatusResourceHealthStatusBeta 1.36, on
PodGroup claimsPodGroup spec.resourceClaimsDRAWorkloadResourceClaimsBeta 1.37, off

Alpha, off : list-type and derived attributes, node allocatable resources, optional node operations, compatibility groups, pool status.

4.Gotchas

  • Preemption never frees a DRA device. A higher-priority Pod stays Pending.
  • spec.nodeName skips the scheduler, so nothing allocates the claim. Pin with nodeSelector on kubernetes.io/hostname.
  • firstAvailable is decided per Pod. Replicas can get different subrequests.
  • Only v1 is on by default. A v1beta1 or v1beta2-only driver needs kube-apiserver --runtime-config.
  • The admin label is resource.kubernetes.io/admin-access="true", not resource.k8s.io.

5.Commands

CommandShows
kubectl get deviceclasses"No resources found" = on
kubectl get resourceslices --field-selector spec.nodeName=<node>One node's devices
kubectl get resourceclaims -Apending, allocated, reserved
kubectl describe resourceclaim <name> -n <ns>First stop, Pending Pod
kubectl get pod <pod> -o jsonpath='{.status.resourceClaimStatuses}'Generated claim names

Try it : dra-example-driver (mock GPUs, gpu.example.com). NVIDIA driver : gpu.nvidia.com, mig.nvidia.com.

In short : The scheduler allocates devices when it places the Pod, the kubelet has the driver prepare them through CDI, and the allocation is freed once no Pod reserves the claim.