BlogEvaluation guide

Private GPU Cloud vs Public Cloud: An Evaluation Guide

Vendors use the term loosely, so start with a definition you can test, then compare dedicated GPU capacity with shared public cloud using evidence and a pilot.

Picture a platform lead at a mid-sized AI company. The security team has flagged that fine-tuning runs on customer data are happening on shared public cloud instances, and leadership wants a “private GPU cloud” by next quarter. Three vendors reply with decks that all use the phrase. One means a dedicated physical cluster. Another means a reserved block of virtual machines on shared hosts. The third means a private network segment in front of the same multi-tenant hardware everyone else uses. The lead realizes the label tells them almost nothing.

This guide is for ML, platform and security leads making that call. It explains what a private GPU cloud should mean in practice, which evidence to ask for, how to weigh it against the elasticity of public cloud GPU instances, and how to close the decision with a pilot and a scorecard. Provider examples are illustrations of what to look for. Use them to see what good evidence looks like.

1. Define “private GPU cloud” before you compare anything

Write your own definition and make every vendor answer against it. A workable one has three parts, and each part should be something you can verify.

  • Dedicated or single-tenant hardware: the physical GPU servers, and ideally the fabric between them, serve only your workloads for the term.
  • Isolation you can verify: the provider can show you how tenancy is enforced, with documents, host-level access or acceptance evidence, beyond a sentence in a sales deck.
  • A named support boundary: a written split of who owns hardware, firmware, drivers, OS, networking, storage and your application stack.

Public cloud GPU instances usually run on shared, multi-tenant infrastructure with broad managed services and elastic instance families. That model suits many teams well. The point of the definition is to make the trade explicit so you can decide which properties matter for each workload.

2. Ask for tenancy and isolation evidence

The biggest practical difference is whether you get dedicated bare-metal nodes or a slice of a shared, virtualized host. Both can be secure. They produce different evidence and different audit stories.

  • Ask whether each node is dedicated to you for the full term, or whether a hypervisor schedules other tenants onto the same host.
  • Ask whether you get root on the physical machine or a guest OS inside a virtual machine.
  • Ask what the provider can show: node serial numbers, inventory records, acceptance test results, or witnessed test reports.
  • Ask how hardware is reassigned between customers, and what happens to firmware and local storage in between.

As one example of what evidence can look like, Pacific’s reserved B300 offer is described as a single-tenant bare-metal cluster backed by witnessed factory, site and integration acceptance evidence. For a deeper comparison of bare metal, virtual machines and owned pods, see our guide to evaluating bare metal GPU cloud against VMs and on-prem pods.

3. Check the network and fabric, beyond the GPU model

Two offers with the same GPU type can perform very differently on multi-node training because of the interconnect. Treat fabric as a first-class line item in your evaluation.

  • Interconnect type between nodes, for example InfiniBand, and whether it is listed per offer or only promised verbally.
  • Whether the fabric is dedicated to you or shared with other tenants’ traffic.
  • Topology and oversubscription, documented in writing for the nodes you will receive.
  • Who can change fabric configuration, and how changes are requested.
  • Storage paths: local NVMe per node, shared filesystems, and the network those ride on.

Pacific’s catalog listings for H100, H200 and B200 nodes state an InfiniBand interconnect, and the reserved B300 Pod 32 reference describes 32 HGX B300 nodes with a dedicated, configurable fabric. Whichever provider you look at, ask for the same level of specificity and then test it in the pilot.

4. Understand the access model and the operations boundary

How you get onto the machines determines how much of the stack your team owns. SSH access to a bare-metal Linux host with NVIDIA drivers and CUDA installed gives you a familiar starting point and full control above the OS. It also means your team handles scheduling, container runtimes, monitoring and patching for your own software unless the contract says otherwise.

Public cloud instances often come with managed services for orchestration, storage and identity. That reduces operational load, and it also ties more of your workflow to that provider’s services. Neither is wrong. Map it honestly.

  • Who replaces a failed GPU or node, and how is a hardware fault reported and tracked?
  • Who owns driver and CUDA version changes after handover?
  • What monitoring does the provider run on hardware health, and what do you see of it?
  • What support channels exist, during which hours, and with what escalation path?
  • Which actions require a ticket, and which can your team perform directly?

Ask for the boundary as a table in the contract or order form. If the vendor cannot produce one, assume the gaps land on your team.

5. Pin down data handling at end of term

Security leads often focus on isolation during the term and forget what happens when it ends. Model weights, checkpoints and training data sit on local disks and shared storage. You need a clear answer on how that data is removed.

  • What happens to local NVMe when the resource ends, and is that stated in writing?
  • How is shared or network storage cleared, and on what schedule?
  • Can the provider supply a record of erasure, and in what form?
  • What is your own responsibility, for example wiping scratch space and rotating credentials before handback?

Pacific’s H100 and B200 listings, for instance, state that local NVMe is erased when the resource ends. Use statements like that as a starting point, then confirm the method and any documentation your auditors will want.

6. Weigh elasticity against commitment

The core trade-off of dedicated capacity is flexibility. Public cloud instance families are built to scale up and down quickly, with the caveat that specific GPU types can be hard to obtain at the moment you need them. Dedicated capacity usually asks for a term commitment in exchange for knowing exactly which hardware you have.

Rather than comparing headline rates, list the cost drivers that change the real number for your workload:

  • Utilization: how many hours per day the GPUs actually run jobs, including queue gaps and debugging time.
  • Term length and whether you can extend, shorten or step between terms.
  • Data movement in and out, and storage you pay for alongside compute.
  • Engineering time to operate bare-metal hosts versus using managed services.
  • Cost of idle capacity if a project slips, and cost of waiting if capacity is unavailable.

Many teams split the decision: steady training and sensitive fine-tuning on committed dedicated nodes, bursty experiments on elastic instances. Pacific’s catalog shows on-demand preview rates for some GPU types and reserved terms of 7 to 30, 31 to 90 and 91 to 180 days by request. Note that preview rates are indicative until the allocation is confirmed, and a paid order is not the same thing as a ready resource. Ask every provider when capacity is actually confirmed. Our breakdown of reserved vs on-demand GPU terms walks through how to model this.

7. Test compliance fit and plan your exit

Compliance needs vary. Some frameworks care most about who can access the data, others about where the hardware sits, and others about documented controls and evidence. Start from your own control list and ask each provider which controls they cover, which they share, and which stay with you.

  • Which regions or sites will host your nodes, and can that be fixed in writing?
  • What physical and logical access controls apply to the hardware?
  • Which reports, attestations or acceptance records can you share with auditors?
  • How are incidents notified, and within what process?

Then plan the exit on day one. Dedicated bare-metal hosts with a standard Linux, driver and CUDA stack tend to make workloads portable, since your containers and scripts do not depend on proprietary services. Heavier use of managed services can make moving harder. Either way, document how you would export data, rebuild environments and move to a different provider or to owned hardware.

8. Close with a pilot and a scorecard

A short pilot on the actual hardware settles this. Run the same representative workload on each finalist and score them against criteria you agreed before the pilot began.

  • Tenancy evidence received and reviewed by security.
  • Fabric behavior on your multi-node job, compared with what was documented.
  • Time from confirmed order to usable SSH access, and accuracy of the environment you received.
  • Quality of support responses to a test ticket.
  • Clarity of the operations boundary and end-of-term data handling.
  • Commercial terms: flexibility, extension options and how pricing is confirmed.
  • Exit effort: how long it took to tear down and move your environment.

Weight each line by what matters most for the workload, and keep notes on anything a vendor promised verbally so it can be added to the contract. If you want a starting template for evaluating dedicated GPU rental options, Pacific’s guide to dedicated bare-metal GPU rental on on-demand and reserved terms lays out the main options, and the Pacific Compute catalog of dedicated bare-metal GPU nodes shows the GPU types, node sizes and terms you can review before requesting.

If you are weighing dedicated, single-tenant capacity against shared public cloud instances and want to walk through tenancy evidence, fabric and terms for your workload, Pacific Intelligent Technologies, Inc. can help you scope it. Book a 30-minute call to review your requirements.

FAQ

Is a private GPU cloud always more secure than public cloud?

Not automatically. Dedicated, single-tenant hardware removes some shared-host concerns and can make evidence easier to collect, but security still depends on access controls, operations, data handling and your own configuration. Public cloud providers invest heavily in isolation too. Judge each option on the evidence it provides for your specific controls.

What is the difference between bare-metal GPU nodes and GPU virtual machines?

With bare-metal nodes you get the whole physical server, usually with root access to the host OS. With virtual machines a hypervisor sits between you and the hardware, and the host may serve other tenants. Bare metal gives more control and clearer tenancy. Virtual machines often integrate more tightly with managed services and scale more easily.

How long should a private GPU cloud pilot run?

Long enough to run a representative multi-node job, file a support ticket, review tenancy evidence and test teardown. Agree the scorecard and success criteria before the pilot starts so the length is driven by what you need to verify rather than by a calendar target.

Continue on the mothership

This satellite stops at the playbook. Transactions, specs, and comparisons live on pacificmachines.com. If the next step is a human, book 30 minutes with Harper.

Book 30 min