Buying or Renting Compute for Machine Learning

Why this question keeps coming up

At some point, almost every team working on machine learning has to decide how to get access to GPUs. You can rent them by the hour from a cloud provider, or you can buy the hardware outright and run it yourself. Both paths lead to the same computation, but they have very different cost structures, risks and operational demands. The decision is rarely obvious, because the sticker price of a GPU and the hourly price of a cloud instance are not directly comparable numbers. One is a lump sum you pay once; the other is a stream of small payments that only continues as long as you keep paying it.

This article builds a framework for thinking about the decision clearly. It does not tell you that cloud is always cheaper, or that buying always pays off faster, because neither is true. The right answer depends on how much you will actually use the hardware, how long you plan to use it for, how your workload fluctuates, and what else you value besides pure dollars per hour. The goal here is to give you the tools to work out your own break-even point, using your own numbers, rather than relying on someone else's rule of thumb.

Two different ways to pay for the same thing

It helps to separate the financial shape of the two options before comparing their costs. Renting compute from a cloud provider is an operating expense, usually shortened to opex. You pay for what you use, when you use it, and the bill scales up and down with your activity. If you stop using the service, the payments stop. Buying hardware is a capital expense, or capex. You pay a large amount upfront to own a physical asset, and after that the asset is yours whether you use it or not. The ongoing costs of owning it, electricity, cooling, maintenance, are much smaller than the purchase price, but they do not disappear just because the GPU sits idle.

This distinction matters beyond accounting. Opex is flexible and reversible: you can try a workload for a month and stop. Capex is a commitment: once you have bought the hardware, the smart move is to use it as much as possible to get your money's worth, because the purchase price is a sunk cost that does not come back if you decide the project is not working out. Keeping this difference in mind will make the rest of the comparison much easier to follow.

What renting compute actually costs

Cloud GPU pricing is not a single number. Providers typically offer several pricing tiers for the same hardware, and the choice between them changes the economics substantially.

On-demand pricing

This is the simplest option: you request an instance, you are billed per second or per hour while it runs, and you can stop it at any time with no penalty. It is also the most expensive per hour, because you are paying for maximum flexibility. On-demand pricing is the right choice for short experiments, unpredictable workloads, or situations where you genuinely do not know yet how much compute you will need.

Reserved or committed use pricing

If you know you will need a certain amount of compute for an extended period, most providers let you commit to one or three years of usage in exchange for a meaningfully lower hourly rate, often somewhere in the range of a third to half off the on-demand price, though the exact discount varies by provider and changes over time, so always check current terms before relying on a specific figure. The trade-off is that you lose some flexibility: if your needs shrink, you are still paying for the commitment, or at least for the baseline you locked in.

Spot or preemptible instances

Cloud providers sell their unused capacity at a steep discount through spot or preemptible instances, often less than half the on-demand price. The catch is that the provider can reclaim that capacity with little or no warning if someone else needs it at the full price. This makes spot instances a good fit for fault-tolerant, batch-style workloads, such as training runs that checkpoint regularly and can resume after an interruption, but a poor fit for anything that needs to run continuously without disruption, such as a production inference endpoint serving live traffic.

The costs that are easy to forget

The headline per-hour GPU price is rarely the whole bill. When estimating the true cost of renting, also account for the following.

• Storage: persistent disks attached to instances, and storage for datasets and model checkpoints, are billed separately and continuously, even when the instance is stopped.
• Data egress: moving data out of a cloud provider's network, for example downloading a trained model or transferring data to another provider, often costs money per gigabyte. Moving data between regions or providers can add up quickly for large datasets.
• Billing granularity and minimums: some instance types or services have minimum billing increments, so very short jobs can cost more per unit of useful work than the headline rate suggests.
• Orchestration overhead: if you need to manage clusters, queues, or autoscaling, there is often an additional managed-service fee on top of the raw compute price, or an engineering cost if you build and maintain that layer yourself.

What owning compute actually costs

The purchase price is just the start

When people compare buying to renting, they often compare the price of a GPU to the hourly cloud rate and stop there. That understates the true cost of ownership considerably. A GPU does not run by itself: it needs a server chassis, a motherboard, CPU, memory, storage and networking to be usable, and all of that has to be purchased, assembled and kept running.

• Electricity: high-end GPUs can draw several hundred watts each under load, and a multi-GPU server draws proportionally more. Electricity is billed continuously for as long as the hardware is powered on, whether or not it is doing useful work.
• Cooling and power delivery: dense GPU hardware generates significant heat and needs adequate cooling and electrical circuits. Retrofitting a room for this, or renting space in a facility that already has it, is a real cost.
• Physical space and redundancy: racks, uninterruptible power supplies, networking switches and backup components all add to the bill, especially once you move beyond a single workstation-class machine.
• Staff time: someone has to rack the hardware, install drivers and software, monitor for failures, replace faulty parts and keep firmware up to date. This is easy to underestimate if the team has not run physical infrastructure before.
• Insurance and risk: owned hardware can fail, be damaged, or simply underperform expectations, and the financial risk sits entirely with the owner.
• Opportunity cost of capital: money spent on hardware is money that cannot be used for anything else, whether that is hiring, research, or simply sitting in a bank account earning interest. This cost is real even though it never appears on an invoice.

Depreciation and obsolescence

GPUs lose value faster than most people expect, for a reason that has nothing to do with wear and tear: newer generations of hardware regularly improve performance per dollar and performance per watt. A GPU that looked like excellent value two years ago can look expensive today, not because it broke, but because a newer card does the same job faster and more cheaply. This means the resale market for used accelerators, while it exists, rarely returns anywhere close to the original purchase price, and owning hardware carries a real risk of being stuck with an asset that is worth much less than you paid for it well before it physically wears out.

Utilization: the number that decides everything

Here is the intuition before the arithmetic. Imagine deciding whether to buy a car or use a taxi everywhere. If you drive every single day, buying a car is almost certainly cheaper over a few years, because the fixed cost of owning it is spread across a huge number of trips. If you only need a car once a month, a taxi is almost certainly cheaper, because the car would spend most of its life sitting in a driveway while you still pay for insurance, parking and depreciation. Compute works the same way. A GPU that runs training jobs around the clock earns back its purchase price quickly. A GPU that sits idle most of the time, waiting for occasional jobs, is an expensive way to buy a small amount of actual computation.

Utilization is the fraction of available time that the hardware is actually doing useful work. It is the single most important variable in this decision, more important than the headline price of either option, because it determines how many hours of real computation you spread your fixed costs across.

A worked example

The numbers below are illustrative, chosen to make the arithmetic easy to follow. Actual GPU prices, cloud rates and electricity costs vary by region, provider and over time, so treat this as a method to apply with your own quotes, not as a current price list.

Suppose a single GPU server, including the card itself, chassis, CPU, memory, storage and networking, costs 15,000 in local currency to buy. Assume it will be useful for three years before it is either replaced or its resale value becomes negligible, and that maintenance, including occasional part replacements and the staff time to keep it running, costs 2,000 per year. Assume the server draws about 1 kilowatt under typical load, electricity costs 0.15 per kilowatt-hour, and cooling and facility overhead add another 20 percent on top of the raw electricity cost.

The function below turns those assumptions into a cost per hour of actual use, for a given utilization fraction. Utilization of 1.0 means the GPU is running useful work one hundred percent of the available calendar time; 0.2 means it is busy twenty percent of the time and idle the rest.

def owning_cost_per_hour(capex, lifetime_years, utilization,
                          power_kw, electricity_price_per_kwh,
                          overhead_fraction=0.2, annual_maintenance=0):
    hours_per_year = 8760
    total_hours = lifetime_years * hours_per_year
    total_fixed_cost = capex + annual_maintenance * lifetime_years
    fixed_cost_per_used_hour = total_fixed_cost / (total_hours * utilization)
    power_cost_per_hour = power_kw * electricity_price_per_kwh * (1 + overhead_fraction)
    return fixed_cost_per_used_hour + power_cost_per_hour

for u in [1.0, 0.5, 0.2, 0.1]:
    cost = owning_cost_per_hour(
        capex=15000, lifetime_years=3, utilization=u,
        power_kw=1.0, electricity_price_per_kwh=0.15,
        overhead_fraction=0.2, annual_maintenance=2000,
    )
    print(f"utilization={u:.0%}  cost per used hour = {cost:.2f}")

Running this gives roughly 0.98 per hour at full utilization, 1.78 per hour at fifty percent utilization, 4.17 per hour at twenty percent utilization, and 8.17 per hour at ten percent utilization. Notice how steeply the cost per used hour rises as utilization falls. This is the core mechanic of owning hardware: the fixed cost of the server does not change whether it runs constantly or sits idle most of the time, so that fixed cost gets divided among fewer and fewer actual hours of use as utilization drops.

Now compare that to a cloud on-demand rate. Suppose, again illustratively, that renting an equivalent GPU on demand costs 2.00 per hour. At full utilization, owning is clearly cheaper, 0.98 versus 2.00. But the break-even point, the utilization level at which owning and on-demand renting cost the same, can be found by solving the equation directly. With these numbers it works out to roughly 44 percent. Above that utilization, buying wins; below it, renting on demand wins. If the team's actual usage pattern is closer to twenty percent, as it often is for a GPU that sits mostly idle between occasional experiments, renting on demand at 2.00 per hour is far cheaper than owning at 4.17 per hour, even though the owned hardware looks like the obvious bargain when you only glance at the purchase price divided by three years of calendar hours.

This is the single most common mistake in these comparisons: dividing the purchase price by the number of calendar hours in its useful life, rather than by the number of hours it will actually be busy. Calendar hours and used hours are very different denominators, and the gap between them is exactly what utilization measures.

Beyond raw dollars: other factors that matter

Cost per hour is the headline number, but several other factors shift the decision in practice, sometimes enough to override what the arithmetic alone would suggest.

Flexibility and elasticity

Workloads in machine learning are rarely steady. A research team might need forty GPUs for an intense two-week push before a deadline and almost none for the following month. A production inference service might see traffic spike ten-fold during a seasonal event. Cloud compute can expand and contract with this kind of demand in minutes. Owned hardware cannot: if you buy for your peak, you pay for idle capacity the rest of the time; if you buy for your average, you cannot handle the peak without renting anyway.

Time to access

Renting a cloud instance can take minutes. Buying hardware involves selecting a configuration, placing an order, waiting for delivery, which can take weeks or longer during periods of high demand for GPUs, and then the physical work of installing and configuring it. If a project needs compute now, that lead time is a real cost even if it never shows up in a spreadsheet.

Data gravity and compliance

If your data already lives inside a particular cloud provider, keeping compute in the same place avoids the cost and friction of moving large datasets, including the egress fees mentioned earlier. Some organizations also face regulatory or contractual requirements about where data can be processed and stored, which can rule out certain cloud regions or, in some cases, push a team toward owning hardware that sits inside a controlled physical location.

Team expertise and operational burden

Running physical hardware well requires skills that a pure machine learning team may not have: rack-and-stack experience, familiarity with server firmware, driver management, and a plan for what happens when a component fails at three in the morning. Cloud providers absorb all of that operational burden in exchange for their margin. For a small team whose strength is modeling rather than infrastructure, that margin can be a reasonable price to pay to avoid distraction from the core work.

Cash flow and budgeting

A large upfront capital purchase can be difficult for a small organization or an early-stage project to justify, even if it is cheaper over three years, simply because the cash has to be available now. Opex-style cloud spending is easier to start small, scale gradually, and cancel if a project does not pan out. On the other hand, some organizations have budget rules that favor capital spending over recurring operating costs, or tax treatments that make depreciating owned equipment attractive; these considerations are specific to each organization's accounting and jurisdiction, so they are worth a conversation with whoever manages the budget before assuming either direction is automatically preferable.

Hybrid strategies that most teams actually use

In practice, few teams pick a single pure strategy. The common patterns tend to combine owning and renting in ways that match the shape of the fixed cost to the shape of the demand.

• Reserved baseline plus cloud burst: own or reserve enough capacity to cover the steady, predictable part of the workload, and use on-demand or spot cloud instances to absorb spikes above that baseline. This captures most of the savings of ownership without paying for peak capacity that sits idle most of the time.
• Colocation: buy the hardware, but place it in a third-party data center that provides power, cooling, networking and physical security for a monthly fee. This avoids the cost and complexity of building your own facility while still owning the asset and getting the lower per-hour cost of ownership at high utilization.
• Buying previous-generation or refurbished hardware: a card that is one or two generations old is often available at a steep discount compared to the latest release, and for many training and inference workloads the older hardware is still perfectly adequate. This lowers the capex side of the break-even calculation considerably.
• Specialized GPU rental providers: alongside the major hyperscale cloud providers, a number of smaller providers specialize in renting GPU capacity, sometimes at lower prices in exchange for a narrower set of features or less polished tooling. These can be worth comparing for workloads that do not need the full breadth of a hyperscaler's services.
• Splitting by workload type: some teams buy hardware for steady production inference, where utilization is predictably high, and rent cloud capacity for research and experimentation, where usage is bursty and unpredictable. This matches each cost structure to the demand pattern it suits best.

A decision framework

When facing this decision for a real project, working through the following questions in order tends to produce a clearer answer than comparing a single cloud price to a single hardware price.

1. Estimate your expected utilization honestly. Look at past usage if you have it, or make a conservative estimate if you do not. Most teams overestimate how continuously they will use hardware before they actually own it.
2. Decide on a realistic planning horizon, typically two to four years for GPU hardware, after which newer generations will likely make the old hardware much less attractive even if it still works.
3. Get a real, all-in quote for ownership: hardware, networking, power, cooling, maintenance and any staff time required, not just the GPU's sticker price.
4. Get real quotes for the cloud options that fit your workload: on-demand for unpredictable use, reserved or committed pricing for steady baseline use, and spot pricing for fault-tolerant batch work.
5. Run the break-even calculation, like the one shown earlier, using your own numbers rather than someone else's illustrative figures.
6. Weigh the non-financial factors: how quickly you need access, how your demand fluctuates, whether your team can operate physical hardware, and any data location or compliance constraints.
7. Consider a hybrid split rather than an all-or-nothing choice, matching steady baseline demand to ownership or reserved capacity, and bursty or unpredictable demand to on-demand or spot cloud instances.
8. Revisit the decision periodically. Cloud prices change, hardware prices fall as newer generations launch, and your own utilization pattern will shift as the project matures.

Common mistakes

• Dividing the hardware purchase price by calendar hours of useful life instead of by actual hours of use, which makes ownership look far cheaper than it will be at realistic utilization.
• Comparing only the GPU's purchase price to the cloud's hourly rate, ignoring the server, networking, electricity, cooling and staff time needed to actually run owned hardware.
• Ignoring data egress and storage fees when estimating cloud costs, which can be a meaningful fraction of the bill for data-heavy workloads.
• Assuming utilization will be high because the team intends to use the hardware constantly, without checking this assumption against actual past usage once a similar system has been running for a while.
• Committing to a long cloud reservation, or buying hardware, before a workload's shape is well understood, locking in a cost structure that does not match actual demand.
• Treating spot or preemptible instances as a drop-in replacement for steady, uninterruptible workloads, when they are only cost-effective for jobs that can tolerate being stopped and resumed.
• Forgetting that owned hardware depreciates in practical value faster than its physical condition suggests, because newer generations regularly improve performance per dollar.
• Failing to revisit the decision after it is made, even though cloud prices, hardware prices and the project's own usage pattern all change over time.

Summary and what comes next

Renting compute from the cloud and buying hardware outright are two different financial structures for getting the same computation done, and neither one is universally better. Renting trades a higher cost per hour for flexibility, fast access and no operational burden. Buying trades a large upfront commitment and ongoing operational responsibility for a much lower cost per hour, but only once utilization is high enough to spread the fixed cost across enough actual use. The break-even point between the two depends on the specific hardware price, the specific cloud rate, the electricity and overhead costs, and, above all, on how much of the time the hardware will actually be busy.

The most useful thing you can do before making this decision for a real project is to measure your own utilization, even roughly, before committing to either path. If you do not yet have that data, start on the rental side, where commitments are small and reversible, and use the early usage patterns you observe to decide whether a later move toward ownership, a reserved commitment, or a hybrid arrangement makes sense. Because prices on both sides change over time and hardware generations turn over every couple of years, it is worth treating this as a decision to revisit periodically rather than one to make once and forget.