Scheduling
Dynamic Resource Allocation
1.Why DRA ?
- Device plugin : only says how many,
limits: nvidia.com/gpu: 2. No overcommit, no sharing between containers, no pick by attribute, config per node. - DRA : says which. The scheduler picks specific devices from the driver's ResourceSlices by CEL on attributes and capacity. The kubelet has the driver prepare them through CDI.
- Status : GA since v1.34, gate locked on since v1.35. v1.37 serves only
resource.by default.k8s. io/ v1
| Aspect | Device plugin | DRA |
|---|---|---|
| Asks with | integer in limits | a ResourceClaim |
| Picks device | kubelet | scheduler, by CEL |
| Inventory | node allocatable | ResourceSlices |
| Sharing | none | containers, Pods |
| Config | per node | per claim |
| Node setup | Allocate RPC | Prepare + CDI |
| Partitions | static | partitionable (beta) |
| Health | drops from count | device taints (GA) |
Both can run. One resource name on one node comes from the plugin or DRA, not both.
2.The object model
Like PVC and StorageClass (Storage) : the claim asks, the class categorizes, the slice is the inventory.
- DeviceClass (cluster) : CEL selectors, optional config. Made by an admin or a driver chart.
- ResourceSlice (cluster) : devices per pool. The driver owns it and overwrites manual edits.
- ResourceClaim : the request. Status has allocation, reservedFor.
- ResourceClaimTemplate : one claim per Pod, deleted with the Pod (Owners and GC).
- DeviceTaintRule (cluster) : taints devices by driver, pool, device. In v1 since v1.37.
3.One GPU, two containers
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata: { name: big-gpu }
spec:
spec:
devices:
requests:
- name: gpu
exactly:
deviceClassName: gpu.example.com
selectors:
- cel:
expression: device.capacity["gpu.example.com"]
.memory.compareTo(quantity("40Gi")) >= 0
---
apiVersion: v1
kind: Pod
metadata: { name: trainer }
spec:
resourceClaims:
- name: gpu
resourceClaimTemplateName: big-gpu
containers:
- name: main
image: ubuntu:24.04
command: [sleep, infinity]
resources: { claims: [ { name: gpu } ] }
- name: sidecar # same GPU
image: ubuntu:24.04
command: [sleep, infinity]
resources: { claims: [ { name: gpu } ] }
4.Wiring rules
- Each
spec.entry has a name and exactly one of resourceClaimName, resourceClaimTemplateName.resourceClaims[] - A named claim must already exist in the Pod's namespace, or the Pod stays Pending.
- The Pod list only makes it available. A container gets the device only if its
resources.claimsnames it.
5.Inside a request
devices.: a name plus exactly, or firstAvailable[] (ordered fallback subrequests).requests[] - exactly : deviceClassName, selectors, allocationMode ExactCount (default, count 1) or All, adminAccess, tolerations, capacity.
- constraints[] : matchAttribute, distinctAttribute. config[].opaque : driver + parameters.
What CEL sees
device.driver, device., and device., a Quantity (use compareTo).
A missing key is an error that aborts allocation, not a skip. Guard with has(), or optional chaining with orValue().
In short : A Pod asks through ResourceClaims that name a DeviceClass. The scheduler picks real devices from ResourceSlices with CEL, and the driver prepares them on the node.
1.From claim to device
- The kubelet plugin publishes ResourceSlices.
- resourceclaim-controller creates claims from templates.
- DynamicResources runs CEL, picks devices first-fit.
- It writes allocation + reservedFor, then binds the Pod.
- The kubelet calls NodePrepareResources (gRPC).
- The driver returns CDI device IDs.
- The runtime injects them into the containers.
- NodeUnprepareResources after the Pod stops.
- Last Pod gone : deallocated, template claim deleted.
The rest of the cycle : Advanced Scheduling.
2.Sharing a device
- Template : each Pod gets its own claim and devices.
- Shared claim : many Pods name one ResourceClaim, 256 at most (reservedFor).
- Containers : one Pod claim listed in several containers.
- How they share at runtime (time-slicing, MPS) is up to the driver and its opaque config.
3.Features in v1.37
| Feature | Key fields | Feature gate | Maturity |
|---|---|---|---|
| Core DRA | v1 kinds, pod spec.resourceClaims | DynamicResourceAllocation | GA 1.34, locked 1.35 |
| Prioritized list | requests[].firstAvailable[] | DRAPrioritizedList | GA 1.36, locked 1.37 |
| Admin access | exactly.adminAccess + namespace label | DRAAdminAccess | GA 1.36 |
| Device taints | devices[].taints[], exactly.tolerations[] | DRADeviceTaints | GA 1.37 |
| DeviceTaintRule | deviceSelector {driver, pool, device} | DRADeviceTaintRules | GA 1.37 |
| Extended resource | extendedResourceName : limits served by DRA | DRAExtendedResource | GA 1.37 |
| Device status | claim status.devices[] | DRAResourceClaimDeviceStatus | GA 1.37 |
| Partitionable devices | sharedCounters[], consumesCounters[] | DRAPartitionableDevices | Beta 1.36, on |
| Consumable capacity | allowMultipleAllocations, capacity.requests | DRAConsumableCapacity | Beta 1.36, on |
| Binding conditions | bindingConditions, bindsToNode | DRADeviceBindingConditions | Beta 1.36, on |
| Resource health | allocatedResourcesStatus | ResourceHealthStatus | Beta 1.36, on |
| PodGroup claims | PodGroup spec.resourceClaims | DRAWorkloadResourceClaims | Beta 1.37, off |
Alpha, off : list-type and derived attributes, node allocatable resources, optional node operations, compatibility groups, pool status.
4.Gotchas
- Preemption never frees a DRA device. A higher-priority Pod stays Pending.
spec.nodeNameskips the scheduler, so nothing allocates the claim. Pin with nodeSelector on kubernetes.io/hostname.- firstAvailable is decided per Pod. Replicas can get different subrequests.
- Only v1 is on by default. A v1beta1 or v1beta2-only driver needs kube-apiserver
--runtime-config. - The admin label is resource.kubernetes.io/admin-access="true", not resource.k8s.io.
5.Commands
| Command | Shows |
|---|---|
kubectl get deviceclasses | "No resources found" = on |
kubectl get resourceslices --field-selector spec. | One node's devices |
kubectl get resourceclaims -A | pending, allocated, reserved |
kubectl describe resourceclaim <name> -n <ns> | First stop, Pending Pod |
kubectl get pod <pod> -o jsonpath='{.status. | Generated claim names |
Try it : dra-example-driver (mock GPUs, gpu.example.com). NVIDIA driver : gpu.nvidia.com, mig.nvidia.com.
In short : The scheduler allocates devices when it places the Pod, the kubelet has the driver prepare them through CDI, and the allocation is freed once no Pod reserves the claim.