Spot GPU interruptions follow three different clocks on the three major clouds, and the drain plan that survives all of them is the one sized to the shortest clock. Amazon EC2 issues its interruption notice two minutes before it stops or terminates a Spot Instance. Azure evicts Spot Virtual Machines with 30-seconds notice, delivered on a best-effort basis through Scheduled Events. On Google Cloud the preemption notice duration defaults to 0 seconds unless you request 120 seconds at instance creation, which means the default contract is: react inside the shutdown window or lose the node mid-request. Batch scoring, evaluation sweeps and asynchronous fine-tuning fit inside these windows; latency-bound interactive serving usually does not.
The notice windows compared
Each provider exposes the interruption through a different surface, so your drain daemon needs one poller per cloud. On AWS the signal appears both as an EventBridge event and in instance metadata: the spot/instance-action key returns the action and its approximate UTC time, and AWS recommends checking every 5 seconds. Microsoft delivers eviction warnings through Azure Scheduled Events, best effort and up to 30 seconds before the eviction, with the eviction policy set to Deallocate by default so the machine can be redeployed later. Google Cloud flips the preempted metadata value to TRUE and then runs any configured shutdown script during a best-effort shutdown period of up to 30 seconds; the optional 120-second notice duration must be chosen when the VM is created.
The practical difference between the clouds is not the discount — Google Cloud quotes discounts of up to 91% for many Spot VM machine types, GPUs and TPUs, and the other providers price spare capacity in the same spirit — it is how much warning the workload gets before power-off:
| Provider | Notice before removal | End state | Detection surface |
|---|---|---|---|
| AWS EC2 Spot | two minutes | terminate, stop or hibernate (set at request time) | IMDSv2 instance-action; EventBridge event |
| Google Cloud Spot | 0 s by default; 120 s optional | STOP (default) or DELETE | preempted metadata key; shutdown script |
| Azure Spot | up to 30 seconds, best effort | Deallocate (default) or Delete | Scheduled Events |
Azure also publishes per-SKU eviction rates, quoted per hour and based on the last 7 days of history, visible in the portal during VM creation and queryable through Azure Resource Graph, which exposes Spot pricing history for the last 90 days and eviction rates for the last 28 trailing days. Use those numbers to rank SKUs before committing a fleet to them: two GPU families with similar on-demand prices can have very different eviction profiles by region.
Draining a GPU node
The drain sequence below assumes one daemon per cloud, idempotent steps, and a queue that already tolerates replica churn. Run it in this order every time, including during scheduled chaos tests.
- Poll the native signal, not a proxy: IMDSv2 every 5 seconds on AWS, the
preemptedmetadata key on GCP, Scheduled Events on Azure. - Fail the readiness probe first. The load balancer stops routing new requests before anything else happens, which turns the drain from a race into a queue.
- Finish in-flight requests or fail them forward to a warm replica; for batch jobs, write partial results to object storage with a resumable marker.
- Checkpoint only what is expensive to rebuild: adapter weights, optimizer state, partial batch results. Never spend the window copying KV cache or compiled graphs — both rebuild faster than they transfer.
- Flush logs, metrics and traces before power-off; a node that dies with buffered telemetry is a node you cannot debug.
Two adjacent pieces of infrastructure make the drain cheaper. A semantic cache in front of the pool absorbs the redirected traffic while replicas rebalance, because cache hits never touch a GPU. And the post-mortem — was the drain fast enough, did any request fail twice — depends entirely on tracing and evaluation data flushed in step five.
Designing the replacement path
Interruption handling is mostly fleet topology. Diversify across instance families and availability zones so a single capacity event takes one slice, not the fleet. On Google Cloud, a managed instance group recreates preempted Spot VMs as soon as resources return, and with instance flexibility the group automatically selects machine types with the lowest observed preemption rates. On Azure, rotate SKUs using the Resource Graph eviction-rate query instead of pinning one size in one region. Keep a capacity floor: a small pool of on-demand or reserved GPU nodes sized to carry interactive traffic alone, with spot as burst capacity above it. That floor is what keeps your p99 honest on the days when several zones reclaim capacity at once.
When spot breaks inference
Google Cloud carries a trap specific to accelerators: Spot VMs with GPUs are preempted by default during maintenance events and cannot automatically restart, so without a managed instance group a maintenance window silently halves your fleet. Google Cloud excludes Spot VMs from the Compute Engine SLA and Azure states plainly that Spot VMs carry no SLA, while AWS documents that a Spot Instance interruption is always possible. If your serving stack pulls model weights at startup, reload time is the number that governs recovery, not the headline notice — size replacement capacity on cold-start time instead. The dividing line most teams land on: spot for batch and asynchronous work, a warm floor for interactive traffic, and a drain path tested by killing nodes on purpose.
Sources
- Amazon EC2 — Spot Instance interruption notices (consulted 2026-11-01)
- Google Cloud — Spot VMs (page dated 2026-10-13; consulted 2026-11-01)
- Microsoft Azure — About Azure Spot Virtual Machines (consulted 2026-11-01)