Spot GPUs Save 70% on Paper. Production Says Otherwise.

Amazon SageMaker’s managed spot training can cut ML training cost up to 90% over on-demand — but only if your job checkpoints. AWS explicitly caps non-checkpointing spot jobs at MaxWaitTimeInSeconds of 3600 seconds, so any LLM fine-tune longer than one hour that cannot resume from a checkpoint is structurally impossible to run on discounted capacity. In 2026, that single constraint decides whether your H100 budget survives the quarter.

The 70% Discount Hides a Tax

Spot pricing for GPU capacity in 2026 looks irresistible on a rate card. An H100 SXM spot node runs roughly $1.35–$1.49/hr versus $2.23/hr on-demand at CoreWeave, and a one-year reserved commitment lands at $1.79/hr, a 20% floor discount (GridStackHub, 2026). The same H100 on AWS spot averages $3.49/hr against an on-demand list well above that (io.net pricing analysis, April 2026). Read those numbers and the answer feels obvious: shovel everything onto spot.

It is not obvious. GPU spot instances on the major clouds carry a 10–30% interruption rate, meaning AWS can reclaim the capacity with a two-minute warning whenever on-demand demand spikes (io.net FAQ; OneUptime, Feb 2026). Every interruption discards every token of work since the last checkpoint. For a 24-hour training run hit three times, io.net’s own worked example shows 2–4 hours of compute thrown away — at $1.53/hr that is $4.59 of pure waste layered on top of the discounted bill (io.net). The 70% headline collapses toward 40% the moment your checkpoint interval is sloppy.

Checkpoint Frequency Is the Real Lever

The savings formula AWS publishes is unsentimental: (1 − BillableTimeInSeconds / TrainingTimeInSeconds) × 100 (AWS SageMaker AI documentation). If you ran 500 seconds but were billed for 100, you saved 80%. The corollary is brutal: every second your job spends re-computing what a previous run already produced is billable time that delivers zero model progress. Checkpoint cadence is therefore the single variable that moves both numerator and denominator.

Practitioner consensus in 2026 lands on a checkpoint every 10–30 minutes, written to durable object storage (S3 or Cloudflare R2), never local NVMe (GridStackHub; distributed training guide, 2026). Below ten minutes, the checkpoint serialization cost — flushing optimizer state, sharded weights, and the data loader cursor for a 70B model can exceed tens of gigabytes — starts to dominate. Above thirty, a single preemption burns too much recompute. The optimum is workload-specific, but it is rarely where teams leave it by default.

For HuggingFace Trainer users, the three lines that matter are save_steps=100, save_total_limit=3, and resume_from_checkpoint=True — the last flag turns a killed job into a paused job (distributed training guide). The same resume-from-checkpoint discipline underpins robust AWS SageMaker deployment for any workload you expect to survive a preemption. On SageMaker, the equivalent is checkpoint_s3_uri plus checkpoint_local_path on the estimator, paired with use_spot_instances=True, max_run, and a max_wait set generously above max_run (OneUptime). Skip the checkpoint URI and SageMaker will refuse to schedule the job beyond the 3600-second cap — a hard guardrail, not a soft suggestion (AWS documentation).

Where Spot Earns Its Keep

Not all workloads benefit equally. Managed spot shines on fault-tolerant training that can pause and resume, batch embedding generation, and offline inference — jobs where a 30-second interruption is a nuisance, not an incident (GridStackHub). A single ml.p3.8xlarge at over $14/hr means a 24-hour run costs roughly $340 on-demand; moving that same job to managed spot with checkpointing routinely recovers 60–90% of the bill (OneUptime). For hyperparameter sweeps, where most trials are disposable anyway, spot is almost a default.

It fails predictably on anything latency-bound. Production inference under 10k requests/day on on-demand beats spot because variable load rewards flexibility; above 50k req/day, reserved capacity wins because predictability guarantees the SLA (GridStackHub). Interactive development sessions and any compliance-governed endpoint that forbids preemption belong on reserved or on-demand, full stop.

The Honest Cost Math

Spot’s true total cost of ownership includes infrastructure most teams forget to budget. Storing sharded checkpoints runs $0.08–$0.12/GB/month on S3, and a single 70B run can accumulate hundreds of gigabytes of optimizer state across save_total_limit generations (io.net). Then there is the engineering overhead: wiring termination-notice handlers, automating instance replacement and checkpoint restoration, and instrumenting the resume path. io.net prices that at 10–20 developer hours, or $1,500–$3,000 at $150/hr (io.net).

For a team running one training job a quarter, that fixed cost can exceed a year of spot savings. For a team running continuous fine-tunes, it amortizes inside a week. The decision is volume-dependent, and most teams misclassify themselves.

Workload shape	Recommended model	Why
Dev / experimentation	On-demand	Flexibility beats cost at low hours
Training < 1 week, fault-tolerant	Spot	Max savings, checkpoint absorbs interruptions
Training > 1 month	Spot or reserved	Spot for savings; reserved if reliability is critical
Inference < 10k req/day	On-demand	Variable load needs flexibility
Inference > 50k req/day	Reserved	Predictable load, guaranteed capacity
Batch embedding / offline inference	Spot	Interruptible, maximizes cost savings

Source: GridStackHub decision framework, 2026 (GridStackHub).

Inference Breaks Before Training Does

The harder problem in 2026 is not training — it is serving. Spot-backed inference endpoints are a trap most teams discover only after a page. When AWS reclaims an H100 mid-decode, in-flight tokens are lost, connections reset, and tail latency explodes for the retry storm that follows. Checkpointing cannot save you here because there is no resumable “job,” only a stream of stateless requests. The 2-minute warning is meaningless when a single request has a 30-second budget.

The engineering pattern that actually works is overprovisioned reserved capacity as a floor with spot as burst: reserve enough H100s to handle p99 latency on the steady-state load, then autoscale spot capacity on top to absorb spikes (GridStackHub). Spot instances that get reclaimed simply shed traffic back to the reserved floor; users see a blip, not an outage. This is the same hybrid discipline well-run web teams have applied to CPU for a decade, and it maps cleanly onto GPU — provided you instrument the failover and your load balancer drains, rather than drops, the reclaimed node.

The related failure mode is spot price volatility during peak hours (roughly 9am–5pm US time), where rates can spike 2–3x and effectively price your burst tier out of existence precisely when you need it (io.net). A burst strategy that ignores time-of-day pricing will look cheap on the monthly report and fragile during every afternoon traffic peak.

Reserve the Floor, Spot the Burst

The mature answer, echoed across the 2026 practitioner guides, is a three-tier portfolio rather than a single bet. Reserve baseline capacity for inference and the training pipelines you cannot afford to lose. Use on-demand for overflow and short experimental runs under a week. Push every fault-tolerant batch job — embedding generation, offline evaluation, hyperparameter trials — onto spot with rigorous checkpointing (GridStackHub; OneUptime). Most providers allow converting on-demand instances to reserved mid-cycle, so the discipline is to validate on-demand for 30–60 days, then commit once utilization data is real (GridStackHub).

The teams that win on GPU cost in 2026 are not the ones chasing the steepest discount. They are the ones who know exactly which workload belongs in which tier, who checkpoint aggressively enough that a preemption costs minutes rather than hours, and who never let an interruption touch a user-facing request. The 70% number is real. It just has to be earned.

Cloud AI