Karpenter autoscales GPU nodes on Amazon EKS by watching for pods that the Kubernetes scheduler cannot place, then provisioning EC2 instances that exactly match each pod’s resource requests, node selectors, and tolerations. Unlike the Cluster Autoscaler, which scales node groups based on aggregate capacity, Karpenter evaluates the scheduling requirements of individual unschedulable pods and launches the cheapest compatible instance type, making it well-suited for GPU inference workloads where cost-per-request and fast node startup matter.
How Karpenter Provisions GPU Nodes
Karpenter monitors pods that the Kubernetes scheduler cannot schedule due to resource constraints, evaluates their scheduling requirements, and provisions new nodes that meet those requirements. For GPU workloads this means a pod requesting nvidia.com/gpu: 1 with a node selector for a GPU instance family triggers Karpenter to launch a compatible accelerated instance rather than a general-purpose node. The controller evaluates resource requests, node selectors, affinities, tolerations, and topology spread constraints before selecting an instance, so the node it provisions will fit the pod on the first attempt.
This pod-driven model eliminates the need to pre-provision GPU node groups or maintain multiple managed node groups for different accelerator types. A single NodePool can span p4d.24xlarge, p5.48xlarge, g6e.48xlarge, and other GPU families, and Karpenter will pick the cheapest available option that satisfies all scheduling constraints at provisioning time. This matters for inference teams running disaggregated serving architectures where prefill and decode pods have different GPU memory profiles.
Configuring NodePools for GPU Instances
A production GPU NodePool should restrict the instance types Karpenter is allowed to provision, apply GPU-specific taints so only workloads requesting accelerators land on those nodes, and set resource limits to cap spend. The table below shows the key fields and their effect.
| NodePool field | Purpose for GPU workloads |
|---|---|
spec.template.spec.requirements | Restrict instance types to GPU families and exclude non-accelerated types |
spec.template.spec.taints | Add an nvidia.com/gpu taint so only GPU-requesting pods schedule onto GPU nodes |
spec.limits | Cap total GPUs Karpenter can provision cluster-wide, preventing runaway spend |
spec.disruption.consolidationPolicy | Control whether Karpenter removes empty or underutilized GPU nodes automatically |
Excluding instance types that do not fit the workload is explicitly recommended in the EKS best practices guide: you use node.kubernetes.io/instance-type with a NotIn operator to prevent Karpenter from ever launching unwanted families. Pinning AMIs to tested versions rather than @latest is also strongly recommended for production clusters to avoid surprise failures when a new GPU driver image ships.
Spot Interruption Handling for GPU
Spot GPU instances can cost a fraction of on-demand pricing, but EC2 reclaims them with as little as two minutes of notice. Karpenter supports native interruption handling that detects spot interruption events, maintenance events, and instance termination signals, then automatically taints, drains, and terminates the affected node ahead of time so a replacement is already running before the instance disappears. You enable this by configuring the --interruption-queue argument with an SQS queue subscribed to EventBridge interruption events.
For inference workloads that cannot tolerate a cold restart, this graceful-drain window is the difference between a transparent migration and a dropped request. The EKS best practices guide advises that pods requiring checkpointing or other graceful draining should enable Karpenter interruption handling in their clusters. Combined with a capacity-aware GPU scaling strategy, spot-backed inference pods can maintain availability during reclamation events while still capturing the cost savings.
Consolidation and Disruption Budgets
Consolidation is Karpenter’s mechanism for actively reducing cluster cost. It identifies nodes that are empty or whose workloads can run on other nodes, removes them, and replaces nodes with lower-priced variants when a cheaper instance type can satisfy the same scheduling constraints. The default consolidation policy is WhenEmptyOrUnderutilized, meaning Karpenter will both delete empty GPU nodes and attempt to replace underutilized ones with smaller or cheaper alternatives.
Disruption budgets let you rate-limit how aggressively Karpenter disrupts nodes. You can set a budget of zero nodes during business hours and allow unlimited consolidation overnight, or cap simultaneous disruptions to protect a minimum number of inference replicas. The karpenter.sh/do-not-disrupt annotation on a pod or node blocks voluntary disruption entirely, which is useful for long-running training jobs that cannot be checkpointed. Note that this annotation does not protect against involuntary disruption from spot expiration or manual deletion.
Karpenter applies a finalizer to every node it provisions, which blocks deletion until the Termination Controller has completed its graceful shutdown sequence: tainting the node, evicting pods through the Kubernetes Eviction API to respect PodDisruptionBudgets, verifying VolumeAttachment cleanup, and only then terminating the underlying NodeClaim in the cloud provider. This prevents orphaned EC2 instances and ensures GPU-attached volumes are properly detached.
Sources
- Amazon EKS Best Practices — Karpenter (AWS Documentation)
- Karpenter Disruption Documentation (karpenter.sh)