GPU Autoscaling by Queue Depth: The Full KEDA Setup

GPU autoscaling for asynchronous inference should key off queue depth, not CPU utilization. KEDA’s aws-sqs-queue trigger reads an Amazon SQS queue and turns its backlog into a replica-count metric, using a per-replica queue target whose default is 5 messages. Combine that with the ScaledObject timing fields and you get a fleet that absorbs bursts in seconds and returns to zero when the queue drains — without ever scraping accelerator-level metrics from inside the pods.

Why CPU Metrics Fail GPUs

A CPU- or memory-based Horizontal Pod Autoscaler is close to useless for inference workers. During forward passes the accelerator is saturated while the pod’s CPU sits mostly idle on orchestration overhead, so utilization-based targets fire late or never. Worse, batched serving stacks keep hardware busy even as per-request latency climbs: an accelerator pegged near full utilization tells you nothing about how long each request waited in line. The queue in front of the workers is the honest signal, because it measures demand that has not yet been served. The same argument applies when several adapters share one GPU — as in multi-LoRA serving — where per-pod utilization is even less meaningful, since one saturated card may be juggling workloads for a dozen tenants.

How KEDA Computes Replicas

The aws-sqs-queue trigger exposes queue depth as an external metric with the AverageValue type: the queueLength value is how many messages one replica is expected to handle. KEDA’s documentation gives the arithmetic directly — if one pod can handle 10 messages, set the target to 10, and 30 messages waiting in the queue scales the deployment to 3 pods. Set activationQueueLength above zero for noise suppression: below that threshold the scaler stays inactive and the workload holds at idleReplicaCount, so a handful of stray events never wakes a GPU node.

What counts as “in the queue” matters just as much as the target. By default, the formula for actual messages adds ApproximateNumberOfMessages and ApproximateNumberOfMessagesNotVisible, because NotVisible in SQS terms means the message is still in-flight and being processed — exactly the work your current pods are already chewing on. If slow consumers distort scaling decisions, set scaleOnInFlight to false; if delayed messages should count toward the backlog, set scaleOnDelayed to true so ApproximateNumberOfMessagesDelayed joins the calculation. These are one-line switches in the trigger metadata, and they change fleet behavior more than any tuning of the target itself.

SQS Metrics Behind the Signal

The numbers KEDA reads originate in Amazon CloudWatch, and AWS states plainly that they are approximate because of SQS’s distributed architecture — close enough for operational decisions, but not a ledger. ApproximateNumberOfMessagesVisible counts only messages currently available for retrieval and processing, which makes it the cleanest proxy for unserved backlog. ApproximateNumberOfMessagesNotVisible covers messages received but not yet deleted or expired — your in-flight work — while ApproximateNumberOfMessagesDelayed hides messages that are not yet eligible for retrieval at all. One caveat from the AWS metrics documentation: on rare occasions NotVisible can report a non-zero value even for an otherwise empty queue, so evaluate consecutive data points before reacting to a single spike. AWS itself recommends ApproximateAgeOfOldestMessage alarms to catch backlog growth that replica counts can hide.

The ScaledObject Checklist

The defaults in a ScaledObject are sensible for web services and aggressive for GPU fleets, so audit each field against your cold-start reality before shipping:

With those fields in hand, the deployment procedure is short: (1) create the queue and grant the scaler’s identity read access to it; (2) define a TriggerAuthentication using pod identity or a role ARN; (3) write the ScaledObject with queueURL, queueLength and awsRegion against the specification reference; (4) set minReplicaCount to cover your worst acceptable cold start; (5) load-test with a synthetic burst and record how long scaling actually takes end to end. Out of the box, a fresh ScaledObject polls triggers every 30 seconds, waits 300 seconds before scaling to zero, and can span 0 to 100 replicas by default — three numbers worth overriding deliberately on GPU nodes that take minutes to become genuinely ready.

Tuning Cooldown and Limits

Underneath KEDA sits the standard Kubernetes HPA. Once replicas exceed zero, the HPA control loop reconciles every 15 seconds by default, and downscale recommendations are held inside a stabilization window that defaults to five minutes — behavior that smooths flapping but also delays cost recovery after a burst. On GPU fleets that delay is mostly harmless; the bigger lever is scale-up latency, where node provisioning, driver loading, and model warm-up routinely dominate. A pod that is Ready can still be minutes away from useful throughput, and KV-cache memory sizing for long contexts shows exactly why warm-up cost deserves its own budget. Add a fallback block — failureThreshold plus a static replica count — so a transient CloudWatch failure degrades to a known floor instead of freezing the fleet, and alert on queue age rather than replica count to catch stuck consumers that a healthy-looking scale will never surface.

Sources

• KEDA documentation — AWS SQS Queue trigger
• KEDA documentation — ScaledObject specification
• Kubernetes documentation — Horizontal Pod Autoscaling
• AWS documentation — Available CloudWatch metrics for Amazon SQS