From LeNet to ResNet: Ideas That Lasted

In the previous part of this path you built a CNN layer by layer: convolutions that detect local patterns, pooling and stride that shrink feature maps, and a final fully connected stage that turns features into class scores. That gives you the parts. This part looks at how researchers actually arranged those parts over roughly two decades, and more importantly, why certain arrangements stuck around while others quietly disappeared.

This is not a museum tour for its own sake. Every modern vision model, including the Vision Transformers you will meet later in this path, reuses at least one idea from the architectures described here: small stacked filters, bottleneck layers, multi-branch processing, or skip connections. Understanding where these ideas came from and what problem they solved makes it much easier to read any new architecture diagram without feeling lost.

Why bother with old architectures

It is tempting to skip straight to whatever architecture is fashionable this year. The trouble is that new architectures are almost never built from scratch. They are recombinations of a small set of ideas, each of which was invented to fix a specific, concrete problem that someone ran into while training a real network on real data. If you know the problem, the idea makes obvious sense. If you only see the final diagram, it looks like an arbitrary pile of boxes and arrows.

So instead of memorising five architecture diagrams, treat this article as five case studies. Each one asks: what broke, what did the authors change, and does that change still show up in networks built today. By the end you should be able to look at an unfamiliar architecture and recognise its ancestry.

LeNet-5: the blueprint (1998)

LeNet-5, developed by Yann LeCun and colleagues for recognising handwritten digits, is the ancestor of every CNN in this article. It was designed for small grayscale images, roughly 32 by 32 pixels, of single digits. Its structure already contains the pattern you learned in part 3: alternate convolution and pooling to build up features, then flatten and use fully connected layers to classify.

Here is roughly how the layers transform a 32x32x1 input, using the shape notation from part 1 of this path (height x width x channels):

• C1: a 5x5 convolution with 6 filters turns the 32x32x1 input into 28x28x6 feature maps
• S2: 2x2 average pooling with stride 2 halves the spatial size to 14x14x6
• C3: a 5x5 convolution with 16 filters produces 10x10x16 feature maps
• S4: another 2x2 average pooling step gives 5x5x16
• C5: a 5x5 convolution with 120 filters collapses the spatial dimension entirely, giving a 1x1x120 vector, which is really acting as a fully connected layer at this point
• F6: a fully connected layer maps 120 values down to 84
• Output: a final layer maps 84 values to 10 class scores, one per digit

The whole network has on the order of sixty thousand trainable parameters, which sounds tiny by today's standards but was already a lot to optimise with the hardware and data available in 1998. LeNet used tanh or sigmoid activations rather than ReLU, and average pooling rather than max pooling, both of which later architectures would replace. But the overall shape, convolution and pooling blocks feeding a small classifier head, is a template that every network in this article still follows in some form.

The lesson from LeNet that lasted is simple but easy to undervalue: local filters applied across the whole image, with shared weights, can learn useful visual features directly from pixels, with no hand designed edge detectors or corner detectors. Everything else in this article is really about scaling that idea up and making it train reliably at larger sizes.

AlexNet: depth meets scale (2012)

Between LeNet and the next architecture in this story, more than a decade passed with relatively little movement in mainstream computer vision toward deep convolutional networks. Two things changed that: much larger labelled datasets, in particular ImageNet with roughly 1.2 million training images across 1000 categories, and GPUs capable of training bigger networks in a reasonable time. AlexNet, from Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, combined both with a handful of training tricks and entered the ImageNet competition in 2012.

The result, widely reported at the time, was a large drop in top-5 error compared to the next best entry that year, roughly fifteen percent versus around twenty six percent for the runner up, figures you will see quoted slightly differently depending on the source but the gap was large enough that it changed the field's direction almost overnight. The architecture itself was five convolutional layers followed by three fully connected layers, taking a 224x224x3 (sometimes quoted as 227x227x3, depending on how the original paper's cropping is interpreted) input image, with around sixty million parameters in total, most of them in the fully connected layers near the end.

What made AlexNet work was not one single trick but several applied together:

• ReLU activations instead of tanh or sigmoid. ReLU is simply max(0, x). It is cheap to compute and, unlike tanh, does not saturate for large positive inputs, which made gradients flow better through deeper stacks of layers and let training converge noticeably faster.
• Dropout on the fully connected layers, randomly zeroing a fraction of activations during training, which reduced overfitting on a network with tens of millions of parameters trained on a dataset that, by today's standards, was still limited in size.
• Data augmentation: random crops, horizontal flips and a form of colour jitter based on PCA of pixel values, effectively multiplying the amount of training data the network saw without collecting a single new image. You will cover augmentation in depth in the next part of this path.
• Training split across two GPUs, because a single GPU at the time did not have enough memory for the full network. This is a hardware detail rather than a conceptual one, but it illustrates how closely architecture choices were tied to the hardware limits of the day.
• Overlapping max pooling (3x3 windows with stride 2, rather than the non-overlapping pooling common before), which gave a small but measurable accuracy improvement over non-overlapping pooling.

Of these, ReLU and dropout are the two ideas that are still everywhere. Almost every network you will train in this path uses ReLU or one of its relatives (such as GELU, which shows up in transformers later in this path) as the default activation, and some form of regularisation inspired by dropout is still common, even though batch normalization, which you will meet shortly, took over much of dropout's original job in convolutional networks specifically.

VGG: depth through simplicity (2014)

AlexNet mixed filter sizes, 11x11 in the first layer, 5x5 later, 3x3 near the end. The VGG networks, from the Visual Geometry Group at Oxford, asked a sharper question: what happens if you use nothing but 3x3 convolutions, stacked many layers deep, with 2x2 max pooling to downsample between stages. The answer was VGG16 and VGG19, named for having 16 or 19 weight layers, and they were, for a while, the architecture most people reached for when they needed strong ImageNet accuracy.

The key insight is about receptive field, the concept from part 2 describing how much of the input image a single output unit can see. A single 5x5 convolution and two stacked 3x3 convolutions end up with the same receptive field, but the stacked version is cheaper and more expressive. Here is the arithmetic, assuming C input and output channels for simplicity:

• One 5x5 convolution: 5 x 5 x C x C = 25C^2 weights
• Two stacked 3x3 convolutions: 2 x (3 x 3 x C x C) = 18C^2 weights
• Three stacked 3x3 convolutions reach the same receptive field as one 7x7 convolution: 3 x (9C^2) = 27C^2 weights, versus 49C^2 for the single 7x7 convolution

Stacking small filters costs fewer parameters for the same receptive field, and because each 3x3 convolution is followed by its own ReLU, the stacked version also gets more non-linear activations in between, which in principle lets the network represent more complex functions with the same receptive field. You can verify the receptive field growth yourself with a short calculation. For a stack of 3x3 convolutions with stride 1 and no pooling in between, each extra layer adds 2 pixels to the receptive field size, starting from the first layer's own 3x3 footprint.

def receptive_field(num_3x3_layers: int) -> int:
    """Receptive field (in pixels) after stacking stride-1 3x3 convolutions,
    with no pooling in between."""
    return 1 + 2 * num_3x3_layers

for n in [1, 2, 3]:
    print(n, receptive_field(n))
# 1 3   -> one 3x3 conv sees a 3x3 patch
# 2 5   -> two stacked 3x3 convs see a 5x5 patch
# 3 7   -> three stacked 3x3 convs see a 7x7 patch, same as one 7x7 conv

This is pure geometry, no library needed, but it is worth running to build intuition: three small filters in a row cover exactly the same ground as one big filter, for less cost.

VGG16 has around 138 million parameters, the vast majority sitting in its fully connected layers at the very end rather than in the convolutional part. That is the architecture's main weakness: it is simple and uniform, which made it easy to understand and to use as a feature extractor for transfer learning (the subject of part 6 of this path), but it is heavy, slow and memory hungry by comparison with what came right after it.

The idea that lasted from VGG is the preference for small, stacked filters over a few large ones. You will see 3x3 convolutions as the default choice in the large majority of convolutional networks built since, including ResNet.

A short detour: batch normalization

Before moving to Inception and ResNet, it helps to mention batch normalization, introduced around the same period as these architectures (associated with an improved version of the Inception network sometimes called BN-Inception). It is not an architecture in its own right, it is a layer you insert after a convolution and before the activation function, and it became such a standard ingredient that every architecture from this point on assumes you will use something like it.

The problem it addresses: as a network trains, the distribution of activations flowing into each layer keeps shifting, because every layer before it is also changing its weights. This makes it harder to pick a learning rate that works throughout training, and it tends to require very careful initialisation. Batch normalization normalises the activations within each mini-batch, for each channel, to have roughly zero mean and unit variance, then applies a learned scale and shift so the network can undo the normalisation if that turns out to be useful. In practice, it lets you use higher learning rates, makes training noticeably less sensitive to initialisation, and often gives a mild regularising effect on top.

You do not need to implement batch normalization from scratch to use it; in PyTorch it is simply nn.BatchNorm2d after a convolutional layer, as you will see in the residual block example later in this article. The idea that lasted is the normalisation layer itself: later architectures swap in variants such as layer normalization or group normalization for different situations, but the principle of normalising intermediate activations during training is now close to universal.

GoogLeNet and the Inception module: going wide, not just deep (2014)

The same year as VGG, a team at Google released GoogLeNet, built around a component called the Inception module. VGG had shown that depth helps, but doubling every filter size and stacking many layers is expensive. Inception asked a different question: instead of picking one filter size per layer, why not run several filter sizes in parallel on the same input and let the network combine them.

An Inception module takes its input and feeds it, simultaneously, into:

• a 1x1 convolution branch
• a 1x1 convolution followed by a 3x3 convolution
• a 1x1 convolution followed by a 5x5 convolution
• a 3x3 max pooling operation followed by a 1x1 convolution

The outputs of all four branches are concatenated along the channel dimension and passed to the next module. The intuition is that objects in images appear at different scales, a face might fill most of a small image or occupy a small corner of a large one, so a single fixed filter size is a poor match for every case. Running multiple filter sizes in parallel and letting training decide how much weight to give each one is more flexible than committing to one size per layer, the way LeNet, AlexNet and VGG all did.

The 1x1 convolutions deserve a closer look, because they solve a real efficiency problem. A 1x1 convolution does not look at any spatial neighbourhood at all, it only mixes information across channels at each pixel. Applied before a more expensive 3x3 or 5x5 convolution, it can reduce the number of channels first, so the expensive convolution operates on a smaller number of input channels. This is often called a bottleneck. For example, if the input has 256 channels and you want to run a 5x5 convolution, running it directly on 256 channels is expensive; inserting a 1x1 convolution that first reduces 256 channels down to, say, 64, then running the 5x5 convolution on those 64 channels, and finally expanding back out as needed, costs far fewer multiplications while losing comparatively little representational power, because most of the useful information in 256 correlated channels can often be captured in far fewer.

GoogLeNet was 22 layers deep, deeper than VGG19, yet had roughly 5 million parameters, an order of magnitude fewer than VGG16's 138 million or AlexNet's 60 million. Two design choices account for most of this saving: the bottleneck 1x1 convolutions described above, and the fact that GoogLeNet replaced the large fully connected layers at the end with global average pooling, averaging each final feature map down to a single number before a small classification layer. That single change removed the overwhelming majority of the parameters that VGG and AlexNet spent on their fully connected layers.

GoogLeNet also used auxiliary classifiers, smaller classification heads attached partway through the network during training, whose losses were added to the main loss with a small weight. The idea was to inject a useful gradient signal into the middle of a fairly deep network, since at the time there was concern that gradients would become too weak by the time they propagated back from a single loss at the very end. These auxiliary branches are discarded at inference time; they only help during training. This same worry, that gradients struggle to reach early layers in a deep network, is exactly the problem that the next architecture solved far more directly.

Three ideas from Inception lasted particularly well: multi-branch parallel processing within a block, the 1x1 convolution as a cheap channel mixer and bottleneck, and global average pooling as a lightweight replacement for large fully connected classifier heads. All three appear, in one form or another, in architectures built well after GoogLeNet, including in some of the efficiency-focused designs described later in this article.

ResNet: let gradients skip ahead (2015)

By 2015, researchers at Microsoft Research, led by Kaiming He and colleagues, ran into something that should not have happened if depth were simply good: a plain convolutional network with more layers performed worse, including on the training set, not just on held out data, than a shallower version of the same design. Since it was worse on the training set itself, this was not overfitting. It was a training difficulty, often called the degradation problem. A network with more layers should, in principle, be able to do at least as well as a shallower one, because the extra layers could always learn to do nothing, that is, learn the identity function. In practice, plain stacks of convolutional layers found it hard to learn an identity mapping, and performance degraded as depth increased.

The fix, now called a residual connection or skip connection, is almost deceptively simple. Instead of asking a block of layers to learn the full output you want, call it H(x), you restructure the block to learn only the difference between the output and the input, F(x) = H(x) - x, and then add the input back: output = F(x) + x. If the best thing a block can do is nothing at all, the network only has to push F(x) toward zero, which turns out to be much easier for gradient descent to achieve than learning an exact identity mapping through a stack of non-linear layers.

The practical effect on training is just as important as the mathematical framing. The addition x + F(x) creates a direct path for the gradient to flow backward through the shortcut, bypassing the convolutional layers entirely. Even if the gradient through F(x) becomes very small deep inside a long network, the identity path still carries a usable gradient signal back to earlier layers. This is why ResNet made it practical to train networks with far more layers than before, the original paper described versions with 34, 50, 101 and even 152 layers, and experimented with versions over a thousand layers deep, something that plain stacked convolutions could not do reliably.

A basic residual block, the kind used in the shallower ResNet variants, looks like two 3x3 convolutions (each followed by batch normalization and a ReLU, except the ReLU after the second convolution is applied after the addition, not before it) with the block's input added to its output. Here is a minimal, runnable version in PyTorch. This assumes PyTorch 2.x; the same code will also work on most recent 1.x releases.

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    """A basic ResNet-style block: two 3x3 convolutions with a shortcut
    connection added before the final ReLU. Assumes the input and output
    have the same number of channels and spatial size."""

    def __init__(self, channels: int):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + identity   # the residual, or skip, connection
        out = self.relu(out)
        return out

# quick check with a toy input, batch of 2 images, 16 channels, 32x32 feature maps
x = torch.randn(2, 16, 32, 32)
block = ResidualBlock(16)
y = block(x)
print(y.shape)  # torch.Size([2, 16, 32, 32])

Notice that the output shape is identical to the input shape, 2 images, 16 channels, 32 by 32 pixels. That is not a coincidence, it is a requirement: you can only add two tensors elementwise if they have the same shape. This creates a practical wrinkle that every ResNet implementation has to handle. Many blocks in a real ResNet also need to change the number of channels or reduce the spatial size using a stride of 2, and in those cases the identity path itself has to be passed through a small 1x1 convolution, with a matching stride, before the addition, so that its shape lines up with F(x). This extra 1x1 convolution on the shortcut path is usually called a projection shortcut, and you will see it described that way in most ResNet implementations.

The deeper ResNet variants, 50 layers and beyond, use a bottleneck block rather than the basic block shown above: a 1x1 convolution that reduces the channel count, a 3x3 convolution that does the main spatial processing on the reduced number of channels, and a final 1x1 convolution that expands the channel count back up, with the shortcut added at the end. This is the same bottleneck trick you saw in the Inception module, reused here to keep a very deep network computationally affordable. ResNet-50, for instance, has around 25 million parameters, notably fewer than VGG16's 138 million despite being considerably deeper and generally more accurate.

ResNet won the ILSVRC 2015 competition and, more importantly for this article, its residual connection turned out to be one of the most widely reused ideas in all of deep learning, not just in vision. The same shortcut pattern, add the input back to the output of a block, appears inside every Transformer block you will study later in this path, in part 9 on Vision Transformers, where it plays exactly the same role: letting gradients reach early layers in a very deep stack without being diluted by dozens of intervening non-linear transformations.

What came after, and which parts of it stuck

ResNet is not the end of the story, though it is a natural place to pause in a path that is building toward transfer learning, detection and transformers. A handful of architectures published in the years after ResNet are worth knowing about briefly, not because you need to memorise their diagrams, but because each one contributed one more idea that shows up in the networks you will use later in this path and beyond.

DenseNet: connect every layer to every later layer

DenseNet pushed the skip connection idea further. Instead of adding a block's input to its output, as ResNet does, each layer inside a dense block receives, as input, the concatenation of the feature maps from every preceding layer in that block, not just the immediately previous one. This encourages feature reuse: early, simple features stay directly accessible to every later layer, rather than having to be re-derived or passed through many transformations. DenseNet models tend to use parameters efficiently, since layers can reuse features rather than relearn them, though the concatenation of many feature maps does increase memory use during training.

MobileNet: separate spatial mixing from channel mixing

MobileNet targeted a different problem entirely: running convolutional networks efficiently on phones and other devices with limited compute and memory. Its key idea is the depthwise separable convolution, which splits a standard convolution into two cheaper steps. A depthwise convolution applies one small filter per input channel independently, handling spatial mixing within each channel but not across channels. A pointwise convolution, which is simply a 1x1 convolution, then mixes information across channels. Doing these two steps separately costs considerably less computation than a standard convolution that mixes space and channels together in one operation, at some cost in representational flexibility that turns out, in practice, to be a reasonable trade for mobile deployment. The 1x1 convolution here is doing the same channel-mixing job it did in Inception and in ResNet's bottleneck blocks, reused once again for a new purpose.

Squeeze-and-Excitation: let the network weigh its own channels

Squeeze-and-excitation blocks, often abbreviated SE blocks, add a small mechanism that lets a network learn how important each channel is for the current input, and rescale the channels accordingly before passing them on. In a squeeze step, each channel's feature map is averaged down to a single number, similar to global average pooling. In an excitation step, a small fully connected network takes those per-channel numbers and produces a weight for each channel, between 0 and 1, which is then used to scale the original feature maps channel by channel. This lets the network dynamically emphasise channels that matter for a given image and suppress ones that do not, at a small computational cost. This is worth remembering because it is an early, specific form of the broader idea of attention, learning to weight different parts of a representation by how relevant they are, which you will meet in a much more general form in part 9 when you study the attention mechanism inside Vision Transformers.

EfficientNet: scale depth, width and resolution together

Every architecture up to this point was scaled up somewhat informally: researchers added more layers, or more channels, or used larger input images, usually based on intuition and experimentation rather than a systematic rule. EfficientNet proposed a more principled approach called compound scaling, which increases depth (number of layers), width (number of channels per layer) and input resolution together, in a fixed ratio determined by a small search, rather than scaling just one of these dimensions on its own. The underlying building block in EfficientNet borrows the depthwise separable convolutions from MobileNet and the channel attention idea from squeeze-and-excitation blocks, which is itself a good illustration of this article's main point: by this stage in the field's development, new architectures were largely recombinations of a small, well tested set of ideas rather than entirely new inventions.

The ideas that lasted, gathered in one place

It helps to step back and list the specific, reusable ideas this article has covered, separated from the specific architectures that introduced them. When you meet a new model later in this path, or anywhere else, try to spot which of these it is using.

• Shared, local filters learned from data, rather than hand-designed features (LeNet). This is the foundational idea behind every convolutional network, including all the others in this list.
• ReLU activations, which avoid the saturation problems of tanh and sigmoid and train faster in deep stacks (AlexNet). Variants such as GELU, used in transformers, follow the same basic reasoning.
• Data augmentation and dropout as ways to get a large, flexible network to generalise despite limited labelled data (AlexNet). You will study augmentation in depth in the next part of this path.
• Small, stacked filters (3x3 convolutions) in place of a few large ones, for a given receptive field at lower cost and with more non-linearity (VGG).
• 1x1 convolutions as a cheap way to mix information across channels, and as a bottleneck that reduces computation before an expensive operation (Inception, reused in ResNet's bottleneck blocks and in MobileNet).
• Multi-branch, parallel processing within a block, letting a network combine information at several scales at once (Inception). Multi-head attention in transformers is, at a conceptual level, a descendant of running several parallel operations and combining their outputs.
• Global average pooling as a lightweight alternative to large fully connected classifier heads (Inception).
• Normalising intermediate activations during training, which stabilises and speeds up optimisation (batch normalization, introduced around the Inception era, now close to universal in some form).
• Skip, or residual, connections that give gradients a direct path back to earlier layers, making very deep networks trainable (ResNet). This exact pattern reappears inside every Transformer block.
• Feature reuse through dense connections between layers (DenseNet).
• Separating spatial mixing from channel mixing to cut computation for resource-constrained settings (MobileNet's depthwise separable convolutions).
• Learned, per-channel weighting of feature importance (squeeze-and-excitation), an early and specific relative of the general attention mechanism used in transformers.
• Scaling a network's depth, width and input resolution together according to a deliberate rule, rather than increasing one dimension in isolation (EfficientNet).

Notice how many of these ideas are not really about accuracy for its own sake, they are about making training work reliably (ReLU, batch normalization, residual connections) or making a given level of accuracy cheaper to achieve (1x1 bottlenecks, depthwise separable convolutions, global average pooling). That is a useful filter to apply whenever you read about a new architecture: ask whether the contribution is primarily about representational power, training stability, or computational efficiency. Most genuinely influential ideas fall cleanly into one of those three categories.

Common mistakes and misconceptions

A few misunderstandings come up often enough when people first study this history that they are worth addressing directly.

• Assuming deeper always means better. ResNet's entire motivation was that plain deep networks were harder to train, not merely more prone to overfitting. Depth helps only when the optimisation problem it creates is also addressed, originally through careful initialisation and, decisively, through residual connections and normalisation.
• Confusing parameter count with computational cost. GoogLeNet has far fewer parameters than VGG16 but is not simply faster in every sense; the right way to compare computational cost is usually FLOPs (floating point operations) or measured wall-clock time on the target hardware, not parameter count alone. A network can have few parameters but still be slow if its operations are not well suited to the hardware it runs on.
• Forgetting that residual connections require matching shapes. If you implement a ResNet-style block and change the number of channels or add a stride inside the main path, you must also adjust the shortcut path, typically with a 1x1 convolution at the matching stride, or the addition will fail, or silently add tensors in a way that does not mean what you intended.
• Treating these architectures as interchangeable defaults regardless of task. A very deep ResNet is unnecessary, and sometimes actually harder to train well, on a small dataset with a simple task; a LeNet-style small network, or a modestly sized architecture combined with transfer learning (part 6), is often a better starting point in practice than reaching for the largest model you have heard of.
• Mixing up training mode and evaluation mode for batch normalization. Batch normalization behaves differently during training, where it uses statistics from the current mini-batch, and during evaluation, where it uses running statistics accumulated during training. Forgetting to switch a PyTorch model to evaluation mode with model.eval() before measuring accuracy on a validation or test set is a common and confusing bug, since it leaves batch normalization using mini-batch statistics for data it

Summary and what's next

LeNet showed that a network can learn its own visual features from raw pixels through shared, local filters. AlexNet showed that given enough data and compute, along with ReLU activations, dropout and data augmentation, that same basic design could scale up dramatically. VGG showed that stacking small 3x3 filters is a cheap, uniform way to grow receptive field and depth together. GoogLeNet introduced parallel multi-scale branches, the 1x1 convolution as a bottleneck, and global average pooling, all aimed at getting more out of fewer parameters. ResNet solved the degradation problem that depth alone had run into, using residual connections that give gradients a direct path through a network, which made truly deep architectures reliably trainable. DenseNet, MobileNet, squeeze-and-excitation blocks and EfficientNet each took one of these ideas further, toward feature reuse, computational efficiency, learned channel weighting and systematic scaling, respectively.

None of these architectures is something you need to reproduce from memory. What matters is recognising their ideas, 1x1 bottlenecks, parallel branches, residual connections, normalisation, when you encounter them inside a new model, a pretrained checkpoint, or a research paper. In the next part of this path you will look at data augmentation, the practical techniques, several of them already mentioned here in the AlexNet section, for getting more effective training signal out of a limited set of images. After that, part 6 will show you how to take one of the pretrained networks from this article, such as a ResNet, and adapt it to a new task through transfer learning, which is, in practice, how most people use these architectures today rather than training them from scratch.