Why this question keeps coming up

Anyone who starts training neural networks quickly runs into the same advice: get a GPU. Tutorials assume you have one, cloud platforms charge extra for them, and job postings list GPU experience as a skill. Yet the reason a graphics card, originally built to render video games, became the main computing engine for deep learning is rarely explained from first principles. This article builds that explanation from the ground up: what a GPU actually is, why its design matches the shape of neural network computation, and where the common misunderstandings lie.

Deep learning is mostly repeated matrix arithmetic

A neural network layer, stripped down to its core operation, multiplies an input vector or matrix by a weight matrix, adds a bias, and applies a nonlinearity. A fully connected layer with 512 inputs and 512 outputs needs 512 by 512 multiplications and additions just to produce one layer's output for one example. Stack a few dozen layers, run millions of training examples through the network, and repeat that forward pass plus a backward pass for every batch, for many epochs, and the total number of multiply-add operations easily reaches the billions or trillions for a realistic model.

Here is a small worked example to make this concrete. Suppose a layer takes a 1 by 3 input vector [1, 2, 3] and multiplies it by a 3 by 2 weight matrix to produce a 1 by 2 output. Each output value is a sum of three products: output_1 = (1*w11 + 2*w21 + 3*w31), and output_2 = (1*w12 + 2*w22 + 3*w32). That is six multiplications and four additions for a tiny layer processing a single example. Now imagine a batch of 256 examples, a layer with 1024 inputs and 1024 outputs, and 50 such layers stacked in a network trained for many iterations. The six multiplications in our toy example become billions of them. The defining fact about this workload is that almost all of those six multiplications in the example, and all of the billions in the real case, do not depend on each other. Computing output_1 does not require knowing output_2 first. This independence is the single most important property for understanding why GPUs help.

CPUs and GPUs solve different problems

A CPU, the central processing unit in every laptop and server, is built around a small number of very capable cores, typically somewhere between 4 and a few dozen in consumer and server machines. Each core is designed to run a wide variety of instructions quickly, make decisions (branching: if this, do that), predict what code will run next, and handle tasks that depend on each other in sequence. A CPU core is like a highly skilled generalist who can cook any dish, adapt the recipe on the fly, and context-switch between many kinds of problems.

A GPU, a graphics processing unit, takes the opposite approach. Instead of a few powerful generalist cores, it packs in a very large number of simpler cores, often numbering in the thousands. Each individual GPU core is much less capable than a CPU core on its own. It is not good at complex branching logic or running many different kinds of tasks at once. What it is extremely good at is doing the same simple operation, such as a multiply and an add, on many different pieces of data simultaneously. This style of computing is called SIMT, single instruction multiple threads, a close relative of the older SIMD (single instruction multiple data) idea.

A useful analogy: a CPU is a handful of expert chefs who can each prepare an entire elaborate dish from scratch, adapting as they go. A GPU is an army of a few thousand kitchen assistants who cannot cook a full meal on their own, but if you need five thousand identical vegetables chopped in the same way, they will finish the job far faster than the five expert chefs working alone. Neural network training is, computationally, overwhelmingly about chopping identical vegetables: applying the same multiply-add pattern across huge arrays of numbers.

Why this matches neural networks so well

Go back to the matrix multiplication example. Each of the output values in that computation is independent of the others, which means they can, in principle, all be computed at the exact same moment if you have enough separate pieces of hardware to do it. A CPU with 8 cores can compute at most 8 of those independent values in parallel at any instant, using the rest of its cleverness on branch prediction and instruction scheduling that a plain matrix multiply does not need. A GPU with a few thousand simple cores can compute thousands of those independent values in parallel at the same instant, even though each individual core is far less sophisticated than a CPU core.

This is the central idea of this whole article: deep learning workloads consist of huge numbers of small, independent, identical arithmetic operations, and a GPU is a piece of hardware purpose-built to execute huge numbers of small, independent, identical arithmetic operations at the same time. The match between the shape of the problem and the shape of the hardware is what produces the speedups that make GPUs the default choice for training neural networks.

Inside a GPU: the pieces that matter

Cores and streaming multiprocessors

A GPU's compute cores are grouped into clusters, commonly called streaming multiprocessors (an NVIDIA term; other vendors use different names for the same general idea). Each cluster contains many simple cores that share some local resources and execute instructions in lockstep on different pieces of data. When you run a matrix multiplication on a GPU, the work is broken into many small chunks, and each chunk is handed to a cluster of cores that processes it in parallel with all the other clusters working on their own chunks.

Tensor cores and mixed precision

Modern GPUs used for deep learning, particularly recent NVIDIA models, include specialized circuits often called tensor cores, which are designed to perform the specific pattern of operation at the heart of neural networks: multiply two small matrices and add the result to an accumulator, all in a single hardware step. These specialized units can be faster and more energy efficient than using general-purpose cores for the same matrix multiplication, especially when working with lower-precision number formats such as 16-bit floating point instead of the standard 32-bit. Training in lower precision, sometimes called mixed-precision training, trades a small amount of numerical accuracy for a meaningful speed and memory advantage, and most modern deep learning frameworks support it as a configurable option rather than a default you need to hand-code yourself.

Memory: VRAM, bandwidth and the PCIe bottleneck

Raw compute power is only half the story. A GPU has its own dedicated memory, usually called VRAM (video RAM), physically separate from the computer's main system memory (RAM) that the CPU uses. Before the GPU can work on your data, your model weights, input batches, and intermediate activations all need to live in VRAM. Two numbers matter a great deal here. The first is capacity: how many gigabytes of VRAM the card has, which sets a hard ceiling on how large a model and batch size you can train before you run out of room. The second is memory bandwidth: how quickly data can move between VRAM and the compute cores, measured in gigabytes per second. A GPU with enormous compute power but a comparatively slow path to its own memory will often sit idle waiting for data, a situation engineers call being memory-bound rather than compute-bound.

There is a third, frequently overlooked bottleneck: getting data from the computer's main system memory onto the GPU's VRAM in the first place. This transfer happens over a connection such as PCIe, and it is drastically slower than the GPU's internal memory bandwidth. If your code repeatedly shuttles small amounts of data back and forth between CPU and GPU, that transfer overhead can dominate the total run time and erase the benefit of having a GPU at all. This is why deep learning code is written to move a batch onto the GPU once, do all the necessary computation while it is there, and only bring results back to the CPU when needed, for instance to log a loss value or save a checkpoint.

Where the GPU fits in a training run

A single step of training a neural network, often called an iteration, has a recognizable shape: a batch of examples is fed through the network layer by layer to produce predictions, called the forward pass; the error between predictions and true labels is measured with a loss function; and then the gradient of that loss with respect to every weight in the network is computed by working backward through the layers, called the backward pass or backpropagation. Both the forward pass and the backward pass are, overwhelmingly, sequences of matrix multiplications, additions, and simple elementwise functions such as applying a nonlinearity. All of these operations have the same independence property described earlier, which is why both passes benefit from GPU parallelism, not just the forward pass.

Batching is the mechanism that lets a GPU show its strength. Instead of processing one training example at a time, deep learning code groups many examples into a batch, say 32, 128 or 512 examples, and runs the forward and backward pass on the entire batch as a single larger matrix operation. A GPU that would only be lightly used processing one example at a time can be kept busy processing hundreds of examples simultaneously, which is exactly the kind of workload it is built for. This is also why increasing batch size, up to the point where VRAM runs out, often improves GPU utilization and training throughput, even though it does not necessarily improve model quality.

Inference, the stage where a trained model is used to make predictions on new data rather than being trained, benefits from the same parallel hardware, though often with smaller batches, sometimes even a batch of one, such as answering a single user's request. GPUs still help here because the per-example computation, a forward pass through many layers of matrix multiplications, is the same kind of workload, just run less often per example than during training.

A small hands-on illustration

The following example uses the PyTorch library, version 2.x, to compare how long a large matrix multiplication takes on a CPU versus a GPU, when a GPU is available. Running this code yourself will show actual numbers for your own machine; no specific timing is quoted here because the result depends entirely on the hardware you run it on.

🐍Python
# Requires: torch 2.x (pip install torch)
# Run on a machine with and without a CUDA-capable GPU to compare.
import torch
import time

size = 4000
a = torch.randn(size, size)
b = torch.randn(size, size)

# Time the multiplication on CPU
start = time.time()
c_cpu = a @ b
cpu_time = time.time() - start
print(f"CPU matmul time: {cpu_time:.3f} seconds")

# Time the same multiplication on GPU, if one is available
if torch.cuda.is_available():
    a_gpu = a.to("cuda")
    b_gpu = b.to("cuda")
    torch.cuda.synchronize()  # make sure the GPU is ready before timing
    start = time.time()
    c_gpu = a_gpu @ b_gpu
    torch.cuda.synchronize()  # wait for the GPU to finish before stopping the clock
    gpu_time = time.time() - start
    print(f"GPU matmul time: {gpu_time:.3f} seconds")
else:
    print("No CUDA-capable GPU available on this machine")

A few details in this code matter. The call to torch.cuda.synchronize() before and after the GPU timing is necessary because GPU operations in PyTorch are launched asynchronously: the Python line a_gpu @ b_gpu returns control to your code almost immediately, while the actual multiplication may still be running on the GPU. Without synchronizing, you would measure only the time it took to queue the work, not the time it took to finish it. The size of 4000 by 4000 is chosen because it is large enough to give the GPU enough independent work to show its advantage; with very small matrices, the overhead of launching work on the GPU and moving data to VRAM can outweigh any benefit, a point worth remembering as a common mistake, discussed next.

Common mistakes and misconceptions

  • Assuming a GPU always makes code faster: for small models, small datasets, or operations with a lot of branching logic and little parallel structure, a GPU can be slower than a CPU once you account for the time spent moving data to VRAM and launching GPU operations.
  • Ignoring memory transfer overhead: repeatedly copying data between CPU and GPU inside a training loop, instead of moving a batch once and keeping computation on the GPU, can dominate run time and hide the GPU's real speed.
  • Judging GPUs by VRAM size alone: a card with more VRAM but lower memory bandwidth or fewer cores is not automatically faster for training; capacity determines what fits, bandwidth and core count determine how fast work gets done once it fits.
  • Expecting perfectly linear speedups from multiple GPUs: splitting work across several GPUs introduces communication overhead between them, so doubling the number of GPUs rarely halves the training time exactly, and the gain depends heavily on the model and how the workload is split.
  • Forgetting that not all deep learning code is GPU-friendly: data loading, text preprocessing, and other steps involving branching or irregular data structures often run on the CPU and can become the bottleneck even when the GPU itself is fast and sitting idle.
  • Overlooking numerical precision: mixed-precision training can speed things up substantially, but it is a deliberate choice with trade-offs, not something that happens automatically just because a GPU is present.

When you do not need a GPU

It is worth saying plainly that a GPU is not always the right tool. Classical machine learning algorithms such as decision trees, random forests, and many implementations of linear and logistic regression are not built around the same large, uniform matrix multiplication pattern that neural networks use, and they often run perfectly well, sometimes faster, on a CPU. Small datasets that fit comfortably in memory and small models with few parameters may train in seconds or minutes on a CPU, making the extra complexity of setting up a GPU environment unnecessary. Early stages of a project, when you are still writing and debugging code, checking that data loads correctly, or testing a model on a handful of examples, are also often better done on a CPU, saving GPU time and cloud costs for the runs that actually need the speed.

Summary and what comes next

A GPU helps deep learning because neural network training and inference consist, at their core, of enormous numbers of small, independent, repeated arithmetic operations, mainly matrix multiplications in the forward and backward pass. A GPU's architecture, thousands of simple cores designed to execute the same instruction on many pieces of data at once, is a close structural match for that workload, which a CPU's smaller number of flexible, general-purpose cores cannot match for this specific kind of task. Specialized tensor cores on modern GPUs accelerate the matrix multiply-accumulate pattern further, while VRAM capacity and memory bandwidth, along with the cost of moving data across the PCIe connection, shape how much of that raw compute power you can actually use in practice.

With this foundation in place, natural next steps are to look at how training is distributed across multiple GPUs in larger projects, how mixed-precision training is configured in a framework such as PyTorch, and how to profile a training loop to find out whether your own code is actually compute-bound on the GPU or bottlenecked somewhere else, such as data loading on the CPU.