GPU autoscaling on Kubernetes breaks for LLM inference because the Horizontal Pod Autoscaler cannot see accelerators and the node layer takes minutes to add one. The fix is a four-layer stack — DCGM exporter, Prometheus Adapter, HPA, and KEDA over Karpenter — because better GPU allocation sharpens placement without touching elasticity. The binding constraint is blunt: it cannot scale a Deployment below one replica, so an idle GPU pod keeps a whole accelerator billed at full rate between requests.
Why Kubernetes Cannot See GPUs
HPA scales on CPU, memory, and custom metrics from the custom.metrics.k8s.io API, and a GPU appears in none of them by default, so a vLLM pod holding a model in VRAM looks idle to CPU and memory probes whether it serves zero requests or fifty. The bridge is NVIDIA’s DCGM exporter, which runs as a DaemonSet on GPU nodes and publishes per-device metrics to Prometheus on port 9400; the dcgm-exporter project documents this exact contract of exposing GPU metrics for Prometheus.
The HPA controller cannot query Prometheus directly, so a Prometheus Adapter must translate each metric into a custom metric the API server understands, and Kubernetes allows only one custom metrics API server per cluster. Without that adapter, DCGM_FI_PROF_GR_ENGINE_ACTIVE — the profiling field that measures the fraction of time the graphics and compute engine is active and is the most reliable proxy for inference saturation — stays locked where HPA cannot reach it.
Node Provisioning and Reclamation
Pod-level autoscaling is useless if the node cannot appear fast enough. The standard Cluster Autoscaler scales node groups, one per GPU SKU, and cannot reclaim a node on GPU utilization alone, so a near-empty accelerator node with any running pod stays up. Karpenter instead manages each instance directly, without node groups, and can retry in milliseconds rather than minutes when capacity is unavailable.
| Dimension | Cluster Autoscaler | Karpenter |
|---|---|---|
| Provisioning unit | Node group per SKU | Direct instance |
| GPU-util reclaim | Not supported | WhenEmptyOrUnderutilized |
| Retry speed | Minutes | Milliseconds |
Reclamation is where most GPU spend leaks. Karpenter’s consolidation controller evaluates three policies — WhenEmpty, WhenEmptyOrUnderutilized, and Balanced — each trading cost savings against pod disruption. A node only becomes a consolidation candidate once it has been stable for the full consolidateAfter duration, because the timer resets whenever a pod is added to or removed from the node, and for jobs that must not be interrupted the karpenter.sh/do-not-disrupt annotation excludes the node from voluntary consolidation entirely.
Scaling Pods on DCGM Metrics
With the pipeline wired, the HPA target should be DCGM_FI_PROF_GR_ENGINE_ACTIVE rather than raw GPU utilization, because raw utilization lumps compute and memory traffic into one number and lags the real saturation point: on transformer inference, utilization above roughly 70 to 75 percent correlates with latency degradation, so an averageValue near 0.70 is the sensible ceiling for a p99 latency SLO.
The same pipeline has a hard floor. HPA’s minReplicas cannot drop below one — the controller simply has no concept of zero — so a Deployment that receives one request an hour still pins one GPU for the whole hour, and the node cannot reclaim it while the pod runs. That floor is why fleets sit at single-digit utilization.
Scale-to-Zero With KEDA
KEDA closes that floor without replacing HPA. It creates and manages an HPA on your behalf and, when every trigger reports zero events, patches the Deployment replica count directly to zero, bypassing HPA entirely. The documented design is a two-phase model — an activation phase that owns the zero-to-one transition and a scaling phase that hands off to HPA for one-to-N — and that split is what makes scale-to-zero possible at all.
For GPU inference the right trigger is rarely CPU. KEDA’s built-in Prometheus scaler queries any PromQL expression directly, so a vLLM deployment can scale on vllm_num_requests_waiting, the waiting-request queue depth, with an activationThreshold that wakes the first replica only when a request queues. GPU utilization can read as moderate while requests pile up, so queue depth is the right signal, with no adapter needed.
Cold Starts and Guardrails
Scale-to-zero is only worth what the wake-up costs, and the wake-up has two stages. KEDA activates a zero-replica Deployment only on the next polling interval, so the first request after scale-to-zero waits up to one pollingInterval before any replica exists, and from there Kubernetes still must schedule the pod, pull the image, and finish initialization. At the node layer the cost compounds: a fresh GPU node with a large model can take eight to fifteen minutes to become request-ready, and pod-only restarts run thirty to sixty seconds.
The guardrails are mechanical. Keep images lean and pre-pull them with a DaemonSet or node image cache; tune KEDA’s cooldownPeriod to 60 to 300 seconds so replicas do not oscillate; and protect the single warm replica with a PodDisruptionBudget of minAvailable: 1 so Karpenter’s consolidation cannot drain your only live endpoint. When KEDA scales a replica down, Kubernetes sends a SIGTERM to signal the intent to terminate, and the pod can delay that termination until it finishes its current request, bounded only by terminationGracePeriodSeconds.
The economics only clear at sustained utilization, and the only way to sustain it without burning idle GPU-hours is an autoscaler that reaches zero and comes back fast. Pair that elasticity with hard GPU isolation and the self-hosted NIM cost math, and a shared cluster stops being a billing liability.