Spot vs Reserved GPU Instances: How to Evaluate the Trade-off

Picture a small ML team that moves a fine-tuning run onto spot GPU instances to stretch its budget. The first few days go well. Then capacity tightens in their region, two nodes are reclaimed mid-epoch, the last checkpoint is hours old, and the run that was supposed to finish before a customer demo slips past it. The hourly savings were real. So was the rework, the weekend of relaunching jobs, and the awkward email to the customer.

This guide is for founders, ML leads and infrastructure owners deciding how much of their GPU footprint can live on interruptible capacity and how much needs to be held for them. It focuses on interruption risk. If your main question is whether to commit to a term or stay on on-demand list pricing, read the companion piece on how to evaluate reserved vs on-demand GPU commitments, then come back here for the interruption side of the decision.

1. What spot and reserved actually mean

Spot GPU instances, also called preemptible or interruptible instances, are spare capacity a provider sells at a discount while keeping the right to take it back. Reclaim notice is usually short, availability shifts by region, GPU type and time of day, and the provider decides when your job stops. Notice behavior and terms differ from one provider to the next, so read the actual documentation for each option you evaluate.

Reserved capacity is held for you for a defined term. You pay for it whether you fill it or leave it idle, and in exchange nobody reclaims it mid-run. Dedicated bare-metal nodes on a reserved term sit at the far end of that spectrum: single-tenant hardware that stays yours until the term ends.

So the useful question in any spot vs reserved GPU decision is how much interruption each workload can absorb, and what each interruption costs when it happens.

2. Sort workloads by interruption tolerance

Start with an inventory of what you actually run in a typical month. Then put each workload into one of two buckets.

Usually tolerates interruption well:

• Checkpointed training and fine-tuning runs that can resume from a recent save with little lost work.
• Batch inference, where each batch is independent and a failed batch can simply be requeued.
• Hyperparameter sweeps, where losing one trial out of many barely moves the result.
• Evaluation jobs that score checkpoints or run benchmarks and can be retried later.

Usually needs uninterruptible capacity:

• Production serving with latency targets, where a reclaimed node means dropped requests or a breached SLA.
• Large synchronized multi-node training, where one lost node can halt every other node.
• Deadline-bound runs tied to a launch, a demo or a contract milestone.
• Regulated data handling, where workloads moving between hosts on short notice complicates access controls, data residency and audit trails.

Some workloads sit in the middle. A mid-size training run with frequent checkpoints may be fine on spot early in a project and risky in the final stretch before a release. Tag those as borderline and revisit them as deadlines approach.

3. Measure the cost of an interruption

The spot discount is easy to see on a pricing page. The cost of an interruption is harder to see, so write it down per workload. Include:

• Lost work since the last checkpoint, counted as GPU-hours across every node in the job.
• Restart and requeue time, including waiting for capacity to come back, reloading data and warming caches.
• Engineer time spent noticing the failure, diagnosing it and relaunching, often outside working hours.
• Missed deadlines and what they cost you: a slipped launch, a delayed customer deliverable, a contract penalty.
• Knock-on effects, such as downstream jobs blocked on a model that isn't finished yet.

Multiply the expected cost per interruption by how often you actually see interruptions. Use your own logs wherever possible, since general averages rarely match your region, GPU type, job size and time of day.

4. Checkpointing and orchestration requirements for spot

Spot only works if a job can lose a node at any moment and continue with minimal rework. Before moving anything over, confirm you have:

• Checkpoint frequency matched to how much work you can afford to lose per interruption.
• Checkpoints written to durable storage off the instance, since local disks disappear with the node.
• A handler that listens for the reclaim signal and triggers a final checkpoint and clean shutdown when there is time.
• Resumable job state: data loader position, random seeds and optimizer state all restored correctly.
• A scheduler that requeues automatically and can fall back to another region, instance type or capacity pool.
• Monitoring that records each interruption with a timestamp, nodes lost and work lost.

If any of these are missing, the first few interruptions become expensive lessons. Build the plumbing before the discount starts to matter.

5. Multi-node jobs and large GPU counts

Synchronized distributed training is where spot gets risky fastest. In data-parallel and model-parallel jobs, every rank waits on every other rank. Lose one node and the whole job stalls or crashes, so a 32-node job carries the reclaim risk of all 32 nodes at once.

Elastic training frameworks can shrink and regrow a job, but they add complexity and often handle tight interconnect topology poorly. If your job depends on a fast fabric such as InfiniBand across a fixed set of nodes, replacement nodes may land in a different placement and run slower than the originals.

Capacity risk compounds this. Getting a handful of spot GPUs is often easy. Getting many co-located GPUs of the same type at the same moment, and keeping them for the whole run, is much harder. The larger the count, the more likely you'll wait for capacity or end up running on a partial cluster. For big runs, ask every provider:

• Can you hold this many nodes of this GPU type in one placement for the full run?
• How is interconnect topology confirmed across those nodes?
• If a node fails, what is the replacement process and who absorbs the lost time?
• How and when is availability confirmed before I commit?

6. Cost drivers: discount versus cost per finished job

Compare options on effective cost per finished job, measured in completed training steps, scored batches or evaluated checkpoints. The hourly rate on its own hides the drivers that decide the real bill:

• Discount depth, and how much it moves with demand in your region.
• Retries and recomputed work after each reclaim.
• Restart overhead: data reloads, environment setup and warmup.
• Engineer time spent babysitting interruptions.
• Idle reserved capacity: hours you pay for and don't fill.
• Term length: longer commitments often improve terms but raise the cost of guessing wrong about demand.
• Storage and data movement for frequent checkpoints.

Spot tends to win when jobs are short, checkpoints are cheap and the schedule is flexible. Reserved tends to win when utilization is high and steady and a single interruption would cost more than the discount saves. For broader context on how GPU type and term shape pricing, see this guide to H100, B200 and B300 GPU rental and pricing factors.

7. Build a blended model

Most teams end up using both. Reserve the baseline that must not stop, then burst or experiment on cheaper interruptible capacity.

• Reserved or dedicated: production inference, flagship training runs, regulated workloads and anything with a hard date.
• Interruptible: sweeps, batch scoring, evaluations and exploratory fine-tunes.
• Size the reserved baseline to the utilization you sustain month after month, and let peaks overflow to spot or on-demand.
• Revisit the split every quarter as models, team size and deadlines change.

For the dedicated side, Pacific's catalog of dedicated bare-metal GPU nodes lists H100, H200 and B200 SXM nodes with 8 GPUs per node and on-demand preview rates, with B300 SXM and GB200 and GB300 NVL72 systems by request. H100 nodes are single-tenant bare-metal Linux with SSH access, NVIDIA drivers and CUDA installed, InfiniBand interconnect, and on-demand plus reserved terms of 7 to 30, 31 to 90 and 91 to 180 days by request. For a fuller H100 checklist, see how to evaluate an H100 cloud rental. Preview rates are indicative, and capacity, the current quote and terms are revalidated at review before you commit. Pacific doesn't sell spot capacity, so pair it with another provider if you want an interruptible tier.

8. Run a short pilot before you commit

Before you sign a term or move production work, run a pilot that produces your own numbers.

• Pick two or three representative workloads: one tolerant, one sensitive and one borderline.
• Run them on spot and on reserved or dedicated capacity in parallel, long enough to see normal swings in demand.
• Log every interruption: time, nodes lost, notice received, work lost and time to resume.
• Log completed work: steps finished, batches scored, evaluations done.
• Record engineer hours spent on recovery and relaunches.
• Calculate effective cost per finished job for each option and each workload.

At the end you'll know which workloads belong on spot, how large the reserved baseline should be, and what the interruption tax looks like in practice for your team.

If you're sizing the non-interruptible side of your plan and want to talk through dedicated bare-metal nodes or a reserved term, the team at Pacific Intelligent Technologies, Inc. can review your workloads and the capacity they need. Book a 30-minute call to review your dedicated GPU capacity plan to get started.

FAQ

Is spot GPU capacity good enough for LLM training?

For smaller fine-tunes and single-node runs with frequent checkpoints, often yes. For large synchronized multi-node training, spot is risky because one reclaimed node can stall the whole job and replacement nodes may land in a slower placement. Many teams run experiments on spot and move flagship runs to reserved or dedicated capacity.

How do I compare spot vs reserved GPU cost fairly?

Compare effective cost per finished job. Add up the GPU-hours you paid for, the work you redid after interruptions, restart overhead and engineer time, then divide by completed output such as training steps or scored batches. For reserved capacity, include the hours you paid for and left idle.

Does Pacific offer spot GPU instances?

No. Pacific offers dedicated, single-tenant bare-metal GPU nodes with on-demand access and reserved terms by request, which fit the baseline of a blended plan. Teams that want an interruptible tier typically source it from a separate provider and keep critical work on dedicated nodes.