From parts to a whole
The previous article took apart the two core operations of a convolutional network: convolution, which slides a small filter over an image to produce a feature map, and pooling, which shrinks that feature map while keeping the information that matters. You saw how padding controls output size, how stride controls how far the filter jumps, and how a single convolution layer turns an image into a stack of feature maps, one per filter.
Those are the parts. This article puts them together into something that can actually look at an image and output a label, such as cat, dog or truck. We will build one concrete network, follow a single image through every layer, track exactly how its shape changes, count its parameters by hand, and see how its output numbers become a decision. By the end you will be able to read any CNN architecture diagram and know what each layer is doing and why it is there. Named architectures such as LeNet, AlexNet and ResNet, and the specific tricks that made them work, are the subject of the next article. Here the goal is to understand the generic recipe that all of them follow.
The anatomy of a CNN: backbone and head
Almost every CNN built for classification has the same two-part shape. The first part is called the backbone or feature extractor. It is a stack of convolutional blocks, each one made of a convolution, a nonlinearity, and usually a pooling step. As the image moves through the backbone, its spatial size (height and width) shrinks while its depth (number of channels) grows. Early layers detect simple patterns such as edges and colour blobs; later layers combine those into more complex patterns, because each later filter looks at a region built from the outputs of many earlier filters.
The second part is called the head or classifier. It takes the final, small, deep stack of feature maps, flattens it into a single long vector of numbers, and passes that vector through one or more fully connected (dense) layers, ending in a layer with one output per class. A function called softmax then turns those final numbers into probabilities that sum to one.
So the full pipeline looks like this: image in, then repeated (convolution, activation, pooling) blocks, then flatten, then one or more dense layers, then softmax, then a predicted class out. Everything in the rest of this article is about understanding each of those steps well enough to build one yourself.
The convolutional block: convolution, activation, pooling
Why we need a nonlinearity
A convolution, on its own, is a linear operation: each output value is a weighted sum of a patch of input values. If you stacked two convolution layers directly on top of each other with nothing in between, the combination would still be mathematically equivalent to a single, slightly larger linear operation. Stacking linear operations never creates anything a single linear operation could not already do. This is exactly the same reason a plain multi-layer network of only matrix multiplications collapses into one matrix multiplication: composing linear functions gives you another linear function.
To let the network represent curved decision boundaries and genuinely complex patterns, every convolution layer is followed by a nonlinear activation function applied independently to each number in the feature map. The standard choice in CNNs is ReLU, short for rectified linear unit, defined simply as the maximum of zero and the input value. If a pixel in the feature map comes out as 3.2, ReLU leaves it at 3.2. If it comes out as -1.7, ReLU replaces it with 0. That is the entire operation: keep positive values, zero out negative ones.
ReLU is cheap to compute, does not squash large values the way older activations such as sigmoid or tanh do, and in practice trains well in deep networks. Its one well-known failure mode, sometimes called a dead ReLU, happens when a filter's weights settle into a state where its output is negative for every input in the training set; the gradient through that unit then becomes zero and it stops learning. This is worth knowing about but is not something you need to engineer around when building your first CNNs. Variants such as Leaky ReLU exist to address it, but plain ReLU remains a perfectly good default.
Why we pool
After the convolution and its activation, a pooling layer (as covered in the previous article, typically max pooling with a 2 by 2 window and stride 2) halves the height and width of the feature map. This does two useful things. First, it reduces the amount of computation and memory needed by every layer that follows, since the feature maps get smaller. Second, it gives later layers a larger effective receptive field: a filter in a deeper layer, even though it is still a small 3 by 3 window, is now looking at a 3 by 3 patch of an already-downsampled map, which corresponds to a much larger region of the original image. This is how a CNN with only small filters throughout can still eventually understand patterns that span most of the image.
Put these three steps together, convolution then activation then pooling, and you have one convolutional block. A CNN backbone is simply several of these blocks stacked one after another, usually increasing the number of filters (and therefore the depth of the feature maps) as the spatial size shrinks.
Watching the shapes change: a worked example
Numbers make this concrete. Suppose we are classifying small colour images that are 32 by 32 pixels, with 3 channels for red, green and blue, into 10 classes. This is the same kind of input size used by the well-known CIFAR-10 dataset. We will build a backbone of three convolutional blocks followed by a small classifier head.
Block 1: a convolution with 16 filters, each 3 by 3, with padding 1 and stride 1, so the spatial size does not shrink from the convolution itself (using the padding formula from the previous article: output size equals input size when padding equals (kernel size minus 1) divided by 2, which for a 3 by 3 kernel is padding 1). This gives feature maps of size 32 by 32 by 16. ReLU is applied next, which does not change the shape. Then 2 by 2 max pooling with stride 2 halves the spatial size to 16 by 16 by 16.
Block 2: a convolution with 32 filters, 3 by 3, padding 1, applied to the 16 by 16 by 16 input. Output is 16 by 16 by 32. ReLU, then pooling, giving 8 by 8 by 32.
Block 3: a convolution with 64 filters, 3 by 3, padding 1, applied to the 8 by 8 by 32 input. Output is 8 by 8 by 64. ReLU, then pooling, giving 4 by 8, sorry, 4 by 4 by 64.
Notice the pattern: the spatial size keeps halving, 32 to 16 to 8 to 4, while the depth keeps growing, 3 to 16 to 32 to 64. This trade-off, trading spatial resolution for channel depth, is one of the most consistent design choices across almost every CNN you will encounter. The intuition is that early on, when the feature maps are large, we only need a few filters to capture simple local patterns like edges and colours. As the maps shrink and each location represents a larger region of the original image, we need more filters to represent the growing variety of combinations those regions can contain.
Counting the parameters by hand
Every convolutional filter has its own set of learnable weights plus one bias term, and this does not depend on the size of the image, only on the kernel size and the number of input and output channels. The formula for the number of parameters in one convolution layer is: kernel height times kernel width times number of input channels, plus one (for the bias), all multiplied by the number of output filters.
For block 1: 3 times 3 times 3 input channels equals 27, plus 1 for the bias equals 28, times 16 filters equals 448 parameters.
For block 2: 3 times 3 times 16 input channels equals 144, plus 1 equals 145, times 32 filters equals 4,640 parameters.
For block 3: 3 times 3 times 32 input channels equals 288, plus 1 equals 289, times 64 filters equals 18,496 parameters.
It is worth pausing on how small these numbers are relative to the size of the images they process. Each convolution layer learns only a few thousand numbers, yet it is applied at every location in the image, which is exactly the weight-sharing property that makes CNNs efficient compared with a fully connected layer that would need a separate weight for every pixel.
From feature maps to predictions: flatten and the classifier head
After the three blocks, we have a feature map of shape 4 by 4 by 64, which is 1,024 individual numbers per image. A dense (fully connected) layer expects a single flat vector of numbers as input, not a 3-dimensional block, so the next step is a flatten operation: the 4 by 4 by 64 block is simply reshaped into a vector of length 1,024, in a fixed, consistent order. Flatten does not involve any learning or computation beyond rearranging numbers; it is bookkeeping, not a layer with weights.
That 1,024-length vector is fed into a dense layer with, say, 128 output units, followed by ReLU, and then a final dense layer with 10 output units, one per class, with no activation applied yet (the activation that turns these into probabilities, softmax, comes next and is usually treated as a separate step).
A dense layer's parameter count is simpler: input size times output size, plus one bias per output unit. The first dense layer has 1,024 times 128 plus 128 equals 131,200 parameters. The second has 128 times 10 plus 10 equals 1,290 parameters.
Add everything up: 448 plus 4,640 plus 18,496 from the convolutions, plus 131,200 plus 1,290 from the dense layers, gives 156,074 parameters in total. Notice something striking: the three convolution layers together account for only about 23,500 of those parameters, while the single flatten-to-128 dense layer alone accounts for over 131,000, more than 84 percent of the whole network. This is a very common pattern in simple CNNs and is one reason flattening a large feature map straight into a dense layer is often the first thing to reconsider when a network overfits or becomes unnecessarily large.
An alternative to flattening: global average pooling
Many modern architectures replace the flatten-plus-dense-layer combination with global average pooling. Instead of flattening the 4 by 4 by 64 block into a 1,024-length vector, global average pooling simply averages every 4 by 4 feature map down to a single number, producing a vector of length 64 (one average per channel), which is then fed directly, or through one small dense layer, into the final classification layer. This removes the large 1,024-to-128 dense layer entirely, cutting the parameter count dramatically and making the network less prone to overfitting, at the cost of throwing away the precise spatial arrangement of features in the final map. We are not building this version here, but it is worth knowing the name, because you will meet it again when the path covers full architectures in the next article.
Turning scores into probabilities: softmax and loss
The final dense layer produces 10 raw numbers, one per class, often called logits. These numbers can be any real value, positive or negative, and they do not sum to anything meaningful on their own. The softmax function converts them into probabilities: it exponentiates each logit (raises the mathematical constant e to the power of that value), then divides each result by the sum of all the exponentiated values, so the outputs are all positive and sum exactly to one.
A small worked example makes this concrete. Suppose for one image, after flattening three of the ten classes for simplicity, the logits are 2.0, 1.0 and 0.1. Exponentiating gives approximately 7.389, 2.718 and 1.105. Their sum is about 11.212. Dividing each by that sum gives probabilities of approximately 0.659, 0.242 and 0.099, which do sum to 1.0 (up to rounding). The class with the highest logit, 2.0, ends up with by far the highest probability, about 66 percent, which matches the intuition that softmax preserves the ranking of the logits while turning them into something you can interpret as confidence.
During training, the network is told the correct class for each image, and we need a single number that measures how wrong its probability output was, so that this number can be reduced through training. The standard choice is cross-entropy loss, computed as the negative natural logarithm of the probability the network assigned to the correct class. If the correct class in our example above is the first one, with assigned probability 0.659, the loss is minus the natural log of 0.659, which is approximately 0.417. If the network had instead been very confident in the wrong class, say assigning only 0.05 probability to the correct one, the loss would be minus the natural log of 0.05, approximately 3.0, a much larger penalty. This is the key behaviour of cross-entropy: it punishes confident wrong answers far more than it rewards a merely correct guess, which pushes the network toward being both correct and well-calibrated.
How the network learns, briefly
This article is about architecture rather than training mechanics, but it helps to know, at a conceptual level, what happens once the forward pass above produces a loss value. Every weight in every convolution filter and every dense layer contributed, in some small way, to the final prediction, and therefore to the loss. A technique called backpropagation uses the chain rule from calculus to compute, for every single weight in the network, how much that weight needs to change to reduce the loss slightly. An optimisation algorithm, commonly a variant of stochastic gradient descent, then nudges every weight a small step in that improving direction. Repeat this process over many images and many passes through the training data, and the filters gradually shift from random noise into detectors of edges, textures and eventually whole object parts. The mechanics of training loops, batching and optimisers belong to a hands-on article later in this path; the point to take away here is simply that nothing in the architecture is hand-designed to detect edges or shapes specifically. The architecture only defines the shape of the computation; the values inside the filters are discovered entirely through this loss-reduction process.
Writing it as code
The architecture described above can be written directly as a PyTorch module. The code below assumes PyTorch 2.x, although it will also run unchanged on recent 1.x versions since it only uses long-standing, stable parts of the API.
import torch
import torch.nn as nn
class SimpleCNN(nn.Module):
def __init__(self, num_classes=10):
super().__init__()
self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, padding=1)
self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, padding=1)
self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1)
self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
self.relu = nn.ReLU()
self.fc1 = nn.Linear(in_features=64 * 4 * 4, out_features=128)
self.fc2 = nn.Linear(in_features=128, out_features=num_classes)
def forward(self, x):
x = self.pool(self.relu(self.conv1(x))) # shape: (batch, 16, 16, 16)
x = self.pool(self.relu(self.conv2(x))) # shape: (batch, 32, 8, 8)
x = self.pool(self.relu(self.conv3(x))) # shape: (batch, 64, 4, 4)
x = x.view(x.size(0), -1) # shape: (batch, 1024)
x = self.relu(self.fc1(x)) # shape: (batch, 128)
x = self.fc2(x) # shape: (batch, num_classes), raw logits
return x
model = SimpleCNN(num_classes=10)
batch = torch.randn(8, 3, 32, 32) # 8 fake RGB images, 32x32 pixels
output = model(batch)
print(output.shape)
total_params = sum(p.numel() for p in model.parameters())
print(total_params)
Running this prints torch.Size([8, 10]), confirming that a batch of 8 images produces 8 rows of 10 logits each, exactly matching the shape arithmetic worked out by hand earlier. The second print statement reports 156074, matching the parameter count computed above exactly, which is a useful way to sanity-check that your understanding of the architecture matches what the framework actually built.
A few details in this code are worth noting explicitly. The forward method is where the architecture is actually expressed as a sequence of operations; the constructor only declares the layers and their shapes, it does not connect them. The output of fc2 is left as raw logits, with no softmax applied inside the model. This is deliberate and matches standard PyTorch practice: the loss function used during training, nn.CrossEntropyLoss, expects raw logits as input and applies the equivalent of softmax and the log internally, in a numerically more stable way than doing it manually. If you wanted the model itself to output probabilities, for example at inference time to display a confidence score, you would apply torch.softmax to the output separately, outside the model.
Design choices that matter
Kernel size
Almost every modern CNN uses small kernels, typically 3 by 3, throughout the network, rather than larger kernels like 5 by 5 or 7 by 7. Two stacked 3 by 3 convolutions cover the same receptive field as one 5 by 5 convolution (since each 3 by 3 layer extends the receptive field by one pixel in each direction, so two of them extend it by two), but with fewer parameters and, importantly, with a nonlinearity inserted between them. Given a choice between one 5 by 5 convolution and two 3 by 3 convolutions, the latter is usually preferred for exactly this reason: more nonlinearity for a similar or smaller parameter budget.
Number of filters per layer
There is no formula that tells you the correct number of filters for each layer; it is a design choice usually guided by experience and experimentation. The pattern of doubling the filter count after each pooling step, as used in the worked example (16, then 32, then 64), is a common and reasonable default, keeping the total amount of computation roughly constant across blocks, since the spatial size shrinks by a factor of 4 (2 by 2 pooling) while the channel count only grows by a factor of 2.
Pooling versus strided convolution
Max pooling is not the only way to shrink a feature map. An alternative is to give the convolution itself a stride greater than one, so that the convolution performs the downsampling as it computes its output, without a separate pooling layer. This removes one operation from the network and lets the downsampling itself be learned rather than fixed, at the cost of one more design decision to make for every layer. Both approaches are used in practice; the previous article in this path covers the stride mechanism itself in detail.
Normalisation and regularisation, briefly
Two components appear in most real CNNs that were deliberately left out of the worked example above, to keep the core shape arithmetic clear: batch normalisation, which rescales the activations inside the network to keep training stable, and dropout, which randomly zeroes out some activations during training to reduce overfitting. Both are implemented as additional layers slotted into the blocks described above (commonly, batch normalisation goes right after the convolution and before the activation). They do not change the overall shape arithmetic you have just learned to do by hand, but they do change how reliably the network trains and generalises. Because they interact closely with the specific architectures covered next, a full treatment is left for the next article and the transfer learning article later in this path.
Common mistakes when assembling a CNN
- Forgetting the activation function between convolution layers. Without it, stacked convolutions collapse mathematically into a single linear operation, and the network loses most of its representational power even though it still has many parameters.
- Letting pooling shrink the feature map below a usable size. If you apply 2 by 2 pooling too many times on a small input, the spatial size can reach 1 by 1 or even attempt to go below that, which will raise a runtime error or silently destroy almost all spatial information before the classifier head sees it.
- Mismatching the flatten size and the first dense layer's expected input size. The in_features of the first Linear layer must equal height times width times channels of the feature map right before flattening; getting this number wrong, especially after changing input image size or adding a pooling layer, is one of the most common sources of shape errors when building a CNN by hand.
- Letting the flatten-to-dense transition dominate the parameter count. As the worked example showed, a single dense layer after flattening can hold the vast majority of the network's parameters, which can make the network much larger, slower and more prone to overfitting than the convolutional backbone alone would suggest.
- Forgetting that CrossEntropyLoss in PyTorch expects raw logits, not probabilities. Applying softmax inside the model and then also using CrossEntropyLoss effectively applies softmax twice, which distorts the loss and slows or damages training; this is a frequent, easy-to-miss bug.
- Not checking that the number of output channels of one layer matches the number of input channels declared for the next layer. Since these are both just integers you type by hand, a mismatch here produces a clear runtime shape error, but it is easy to lose track of as a network grows past three or four blocks.
Summary and what comes next
A CNN built for image classification is a backbone of repeated convolution, activation and pooling blocks, followed by a head that flattens the final feature map and passes it through one or more dense layers to a softmax output. Through one worked example, a 32 by 32 by 3 image passing through three convolutional blocks into a small classifier, you have seen exactly how spatial size and channel depth trade off against each other, how to count parameters in both convolution and dense layers by hand, how softmax turns raw scores into probabilities, how cross-entropy loss measures how wrong those probabilities were, and how all of this maps directly onto a short, complete PyTorch module whose printed shapes and parameter count match the arithmetic exactly.
What you have not yet seen is why specific historical architectures look the way they do, how very deep networks avoid the problems that come with stacking dozens of these blocks, and what tricks like residual connections and batch normalisation actually contribute. That is the subject of the next article in this path, From LeNet to ResNet: Ideas That Lasted, which walks through the architectures that established the patterns you have just learned to build from scratch.
Comments (0)
No comments yet. Be the first to share your thoughts.