Use Dynamic Resource Allocation (DRA) for new Kubernetes AI scheduling work, but do not treat it as a complete platform replacement. DRA solves hardware selection and allocation; inference-aware routing, queue policy, cache locality, and workload orchestration still require separate design decisions. The safest migration is incremental: establish structured GPU claims first, then move inference traffic and scheduling policy behind measurable SLOs.
Kubernetes v1.34 made the core DRA APIs stable and available by default, giving platform teams a supported base for GPU, TPU, NIC, and other specialized-device allocation.
Why Scheduling Changed
Traditional Kubernetes resource requests are adequate when a workload needs a quantity of CPU or memory. They become lossy when a model needs a specific GPU memory capacity, accelerator generation, MIG profile, or interconnect relationship. Treating all devices as interchangeable can leave capacity stranded even when aggregate GPU utilization looks acceptable.
The legacy Device Plugin API exposes accelerators as opaque quantities, which means a request such as one GPU does not express topology, partitioning, or hardware attributes.
DRA changes the unit of scheduling from an integer to a structured claim. A workload can describe the device characteristics it needs, while a driver advertises those characteristics and performs the allocation handshake. This separates the user-facing requirement from vendor-specific implementation details.
Kubernetes describes DRA as a mechanism for more powerful selection, allocation, sharing, and configuration of specialized devices, including GPUs, TPUs, and NICs.
| Decision area | Legacy device plugin | DRA-oriented design |
|---|---|---|
| Request | Opaque device count | Structured ResourceClaim |
| Placement | Labels and scheduler workarounds | Driver-published device attributes |
| Partitioning | Often preconfigured on nodes | Declared through resource requirements |
| Migration | Existing workloads remain valid | Adopt by workload class |
The practical implication is not that every existing Deployment needs rewriting. It is that GPU capacity should become an explicit platform contract. New model-serving templates should expose claims, device classes, and compatibility constraints instead of hiding hardware assumptions in node labels.
DRA Changes the Contract
DRA is best understood as four connected operations: model hardware, request it, schedule it, and actuate the allocation. The separation matters because a scheduler can only make useful decisions when the driver publishes enough information about available devices.
The Kubernetes Device Management Working Group describes this flow through ResourceSlice for hardware modeling, ResourceClaim for workload requests, scheduler matching, and actuation before the Pod uses the device.
For platform engineers, this creates a new contract between infrastructure and application teams. The platform owns the device classes and permitted selectors; application teams state requirements such as memory, accelerator family, or a compatible partition profile. Admission policy should reject claims that exceed tenant quotas or use unsupported attributes.
DRA graduated to General Availability in Kubernetes v1.34, with stable resource.k8s.io APIs including ResourceClaim, DeviceClass, ResourceClaimTemplate, and ResourceSlice.
That maturity does not eliminate driver-specific behavior. A production readiness review should verify driver version, supported accelerator families, CDI integration, failure reporting, and the behavior of claims during node drain or autoscaler replacement. DRA is an API foundation, not proof that every vendor implementation supports the same operational features.
The migration boundary should therefore be explicit. Keep the device plugin for unchanged workloads, deploy DRA beside it where supported, and compare allocation success, pending time, fragmentation, and recovery behavior before expanding the claim-based path.
Inference Needs Different Routing
GPU placement is only the first scheduling decision for an LLM service. Once a request enters a serving pool, round-robin routing can send it to a replica without the relevant prefix cache, with a saturated queue, or with an unsuitable model variant. The result is avoidable prefill work and less predictable latency.
llm-d documents a Kubernetes-native serving stack that adds LLM-aware load balancing, distributed KV caching, disaggregated serving, multi-node execution, and autoscaling across model-server Pods.
The key distinction is between infrastructure scheduling and request routing. DRA answers which hardware a Pod receives. An inference router answers which ready replica receives a particular request. Those decisions should share telemetry, but they should not be collapsed into one control loop.
Gateway API Inference Extension defines an inference router that selects an endpoint using model-serving metrics and capabilities, including prefix-cache status and adapter availability.
Prefill and decode also have different resource profiles. Prefill processes the prompt and tends to benefit from compute throughput; decode generates tokens repeatedly and is sensitive to memory bandwidth, KV-cache residency, and queueing. Separating the phases can make independent scaling possible, but it adds cache-transfer, topology, observability, and failure-recovery requirements.
llm-d describes prefill/decode disaggregation as moving KV cache between dedicated workers over high-speed interconnects so time-to-first-token and time-per-output-token can be managed separately.
Do not adopt disaggregation because a benchmark claims a universal speedup. Measure prompt-length distribution, cache-hit rate, interconnect bandwidth, first-token latency, output-token latency, and cost per request under your traffic mix. A single homogeneous model with short prompts may gain more from continuous batching than from a second serving tier.
Migration Path for Teams
A controlled migration should change one contract at a time. Start with a representative inference workload that has clear GPU requirements and stable traffic, rather than a distributed training job whose scheduling behavior can obscure DRA-specific failures.
A ResourceClaim can express a workload’s hardware requirement independently of the Pod template, allowing the platform to evolve device discovery without forcing every application to encode vendor-specific node labels.
- Inventory. Record accelerator models, memory, topology, MIG configuration, drivers, device plugins, node pools, and tenant ownership.
- Define classes. Create a small set of approved DeviceClass and claim patterns; avoid exposing arbitrary selectors to application namespaces.
- Shadow. Run claim resolution and placement checks against production-like workloads without moving live traffic.
- Canary. Migrate one model-serving pool and compare scheduling latency, GPU fragmentation, request latency, error rate, and rollback time.
- Expand. Move workload classes in order of operational value, retaining a tested device-plugin fallback until recovery drills pass.
Kubernetes v1.34 also improved scheduler requeue behavior through plugin-specific callbacks, a change that can reduce unnecessary retries when dynamic resource allocation is in use.
Use the existing internal analysis on DRA and Kubernetes GPU scheduling as background, but keep the migration decision tied to measurements from your own cluster. The article on prefill and decode disaggregation is useful context for the serving-side trade-offs, not a substitute for a workload benchmark.
Operational Checks That Matter
After the first canary, review the system as a set of interacting control loops. Device allocation, Pod scheduling, model loading, request routing, autoscaling, and quota enforcement can each appear healthy while the end-to-end service degrades.
The Gateway API Inference Extension request flow selects an InferencePool, obtains endpoint metrics, asks the picker for a destination, and then routes the request to that endpoint.
That flow makes telemetry a scheduling dependency. Export claim allocation failures, pending duration, device-class matches, model readiness, queue depth, prefix-cache locality, KV-cache pressure, time to first token, time per output token, and GPU memory utilization. Break down each metric by model, namespace, accelerator class, and tenant.
llm-d uses queue depth and request metrics for Kubernetes-native autoscaling patterns, while its router can use cache locality and available memory when selecting a model-server replica.
- Can a tenant request only approved accelerator classes?
- Does a failed claim produce an actionable event rather than a silent Pending Pod?
- Can the serving layer distinguish model readiness from container liveness?
- Can traffic be drained without discarding reusable KV cache unnecessarily?
- Can training be queued or preempted without violating inference SLOs?
- Can the team revert to the previous allocation path within one release?
Set rollback thresholds before the canary: for example, a sustained increase in scheduling latency, a higher rate of unschedulable claims, worse cache-hit rate, or a regression in p95 first-token latency. The exact thresholds belong to the service owner, but the decision must be automatic enough to prevent a migration from becoming a debate during an incident.