Spot GPU instances can reduce AI training costs by 60 to 90 percent compared to on-demand pricing across AWS, Google Cloud, and Azure. The savings are real, but they carry a structural risk: the cloud provider can reclaim the instance with as little as two minutes of warning, terminating a training job mid-epoch. As of early 2026, H100 8-GPU on-demand instances cost approximately $55 to $60 per hour on AWS, $80 to $90 on Google Cloud, and close to $98 per hour on Azure in U.S. regions, while spot equivalents drop per-GPU costs to the $2 to $6 range. Whether the tradeoff pays off depends on checkpoint frequency, data pipeline reproducibility, and how much wall-clock time a restart actually costs.
How Spot GPU Pricing Works
Cloud providers auction spare GPU capacity at dynamic prices that fall well below on-demand rates. The discount is substantial: spot and preemptible pricing can reduce hourly GPU costs by up to roughly 90 percent on AWS, 60 to 91 percent on Google Cloud, and 80 to 90 percent on Azure. The mechanism differs slightly by provider. AWS Spot Instances can be interrupted with a two-minute warning. Google Cloud preemptible VMs terminate after a maximum of 24 hours. Azure spot VMs are evicted based on capacity demand. The economic principle is the same across all three: you trade guaranteed availability for a lower price.
For H100 instances specifically, the per-GPU spot rate lands between $2.00 and $6.00 per hour depending on provider and region, compared to $6.50 to $13.00 on-demand. An AWS p5.48xlarge with eight H100 GPUs at spot pricing costs roughly $16 to $24 per hour for the full instance versus $52 to $56 on-demand. That gap is why teams building fault-tolerant training pipelines gravitate to spot: the hourly savings compound across multi-day runs, and the infrastructure patterns needed to handle interruptions (frequent checkpointing, resumable data loaders, queue-based job scheduling) are the same patterns that improve reliability generally.
The picture shifts for smaller models. AWS only sells the A100 in an 8-GPU configuration (p4d.24xlarge), so you pay for all eight accelerators even if your job uses one. Google Cloud and Azure both offer single-GPU A100 options, which matters when fine-tuning models under 13 billion parameters where one GPU is sufficient. A committed-use discount on a single-GPU A100 from Google Cloud or Azure can cost less than the cheapest AWS spot A100 for that class of workload.
When Spot Training Pays Off
Spot pricing favors workloads where the cost of a restart is bounded and predictable. The decision breaks down along three axes: run duration, checkpoint frequency, and dataset reproducibility. A training job that checkpoints every 500 steps and can resume from the last saved state in under two minutes loses negligible progress to an interruption. A job that saves checkpoints every 10,000 steps on a multi-day run can lose hours of compute. On a 64-GPU cluster at $40 per hour spot, losing six hours means a $9,600 waste event that may exceed the on-demand premium for the same period.
| Provider | Instance | On-demand per GPU/hr | Spot per GPU/hr | Max discount |
|---|---|---|---|---|
| AWS | p5.48xlarge (8x H100) | $6.50 to $7.00 | $2.00 to $3.00 | ~90% |
| Google Cloud | a3-highgpu-8g (8x H100) | $9.00 to $11.50 | $2.50 to $4.00 | ~91% |
| Azure | ND96isr H100 v5 (8x H100) | $11.00 to $13.00 | $3.50 to $6.00 | ~90% |
| AWS | p4d.24xlarge (8x A100) | $2.70 to $2.80 | ~$0.80 to $1.00 | ~70% |
Provider choice matters as much as pricing model. Google Cloud applies automatic sustained-use discounts of up to 30 percent without any commitment, while AWS rewards spot users with the deepest maximum discounts. Azure sits between the two, with enterprise agreement pricing that does not appear on any public rate card. For teams in Portugal and the EU, region availability and data residency add another constraint: not all instance types are available in EU regions, and spot capacity in smaller regions can be tighter than in us-east-1.
Handling Spot Interruptions
Spot instance interruptions can waste hours of training progress if the checkpointing strategy is not solid, and that cost does not appear anywhere on the hourly GPU bill. The mitigation is straightforward but requires discipline: save optimizer state, learning rate scheduler state, and the current data loader position at every checkpoint, not just model weights. PyTorch Lightning and Hugging Face Trainer both support resumable checkpoints natively. The key is to set the save frequency high enough that restart cost is bounded to a small fraction of total run time.
A practical checklist for spot-resilient training:
- Checkpoint every N steps where N is small enough that a restart costs under 5 percent of total run time.
- Store checkpoints to durable, low-latency storage such as local NVMe or a shared NFS mount that survives instance recycling.
- Use a job orchestrator (Ray, Kubernetes with Volcano, or SageMaker Training Jobs) that detects termination signals and requeues automatically.
- Make data loading deterministic and resumable: shard the dataset, record the global step, and skip processed shards on restart.
- Monitor interruption frequency per region and instance type; some regions reclaim spot capacity far more aggressively than others.
Training frameworks that maximize throughput per GPU-hour also reduce total wall-clock exposure to interruptions. For a deeper look at how memory paging translates to concrete throughput gains on identical hardware, see the PagedAttention throughput settings guide.
Inference on Spot GPUs
Inference workloads are a different proposition. A training job can resume from a checkpoint; an inference endpoint serving live traffic cannot easily hand off in-flight requests when its GPU disappears. For this reason, spot GPUs are better suited to batch inference, offline evaluation, and background processing than to latency-sensitive serving. When latency does not matter, spot can dramatically lower inference cost per token.
NVIDIA reports that its NIM inference microservices, running Llama 3.1 8B on a single H100 with FP8, achieve 1,201 tokens per second with NIM enabled versus 613 tokens per second without. Higher throughput per GPU-hour means fewer spot instances are needed to meet batch demand, and the savings compound with the spot discount itself. For teams weighing whether to build on managed cloud AI platforms versus self-hosting on spot capacity, the broader infrastructure and compliance picture for EU-based teams is covered in the practical guide to cloud AI in Portugal.