Transfer Learning and Fine-Tuning

Why train from scratch when you don't have to

In part 4 you saw how architectures like LeNet, AlexNet, VGG and ResNet grew deeper and more capable over time, and how ResNet in particular made it possible to train networks with dozens or even hundreds of layers. Training one of those networks well, from random initial weights, took a very large labelled dataset and a lot of compute. The ImageNet dataset commonly used for this has roughly 1.28 million training images spread across 1000 classes, and even with modern hardware, training a ResNet on it from scratch takes many GPU-hours.

Most real projects do not look like ImageNet. If you are building a classifier for defective solder joints, bird species in a local park, or skin lesions from a dermatology clinic, you might have a few hundred or a few thousand labelled images, not millions. Training a deep CNN from scratch on that little data almost always ends badly: the network has far more parameters than the data can constrain, so it memorises the training set and generalises poorly to new images. You learned in part 5 that data augmentation helps stretch limited data further, but augmentation alone cannot manufacture the millions of examples needed to learn good filters from nothing.

Transfer learning sidesteps this problem. Instead of learning visual features from zero, you start from a network that has already learned them on a large dataset, and you adapt that knowledge to your smaller, more specific task. This article explains what actually transfers between tasks, the vocabulary used to describe different transfer strategies, how to decide which strategy fits your situation, and the practical mechanics of freezing, unfreezing and fine-tuning a pretrained network.

The core intuition: features form a hierarchy

In part 3 you built a CNN layer by layer and saw that each convolutional layer extracts features from the output of the previous one, forming a hierarchy. In part 4 you saw this hierarchy become visible in practice: the earliest convolutional layers of a trained network tend to detect edges, colour contrasts and simple textures, regardless of what the network was ultimately trained to classify. Layers a bit deeper combine those into small motifs such as corners, stripes or simple curved segments. Deeper still, layers respond to parts of objects such as wheels, eyes or leaves. Only the last few layers assemble these parts into representations tied to specific classes, such as "this combination of parts looks like a golden retriever rather than a wolf".

The useful observation is that the early and middle layers are not really about golden retrievers or wolves at all. They are about generic statistical regularities of natural images: edges appear at certain orientations, textures repeat, object parts have characteristic shapes. These regularities show up in almost any photograph, whether it is a dog, a car, a flower or an X-ray of a lung. A network that has learned to detect them on one large dataset has learned something broadly reusable, not something narrowly tied to the 1000 categories it happened to be trained on.

This is the entire premise of transfer learning: take a network trained on a large, general dataset, keep the feature-extracting layers it already learned, and only adapt the parts of the network that are specific to your new task. You are not starting from nothing; you are starting from a network that already knows how to see, and you are teaching it to recognise your particular categories.

What actually transfers, and what does not

Early convolutional filters that detect edges, blobs and colour gradients transfer almost universally, because every natural image is built from these primitives. Mid-level filters that detect textures and simple shapes also transfer well across most everyday photographic domains. What transfers less reliably is the very end of the network: the final fully connected layers (or final convolutional block in fully convolutional designs) that combine high-level features into a decision about the original 1000 ImageNet classes. Those layers are specific to the source task's label set and are usually the first thing you replace, not reuse.

There is also a question of domain, not just task. A network trained on everyday colour photographs will transfer features well to another everyday colour photograph task, even one with completely different classes. It will transfer less cleanly to grayscale medical scans, satellite imagery, or microscope slides, because the low-level statistics of those images (contrast patterns, typical object scales, colour distributions) differ from natural photographs. Transfer learning still usually helps in these cases, often considerably, but you should expect to do more adaptation, and the gains tend to be smaller than when the domains are close.

Vocabulary: pretraining, transfer learning, feature extraction, fine-tuning

These terms are often used loosely, so it helps to pin down precise meanings before going further.

• Source task and target task: the source task is what the network was originally trained to do (for example, classifying ImageNet's 1000 categories). The target task is your actual problem (for example, classifying five species of flower).
• Pretrained model: a network whose weights were already learned on the source task, typically distributed by a library or research group so you do not have to train it yourself.
• Transfer learning: the umbrella term for reusing knowledge learned on the source task to help with the target task. It covers several concrete strategies.
• Feature extraction: a transfer learning strategy where you freeze the pretrained layers (their weights never change) and only train a new, usually small, head on top, using the frozen layers purely as a fixed feature computer.
• Fine-tuning: a transfer learning strategy where, in addition to training the new head, you also continue updating some or all of the pretrained layers' weights, usually with a small learning rate, so they adjust slightly toward your target task.

Feature extraction and fine-tuning are not two unrelated techniques; they sit on a spectrum. At one end, every pretrained weight is frozen and only the new head learns. At the other end, every weight in the network is updated, including the earliest convolutional filters, just starting from a much better initial point than random weights. In between, you can unfreeze some layers and leave others frozen. Choosing where on this spectrum to sit is the main practical decision in transfer learning.

A decision matrix: dataset size and domain similarity

Two questions determine where on the feature-extraction-to-fine-tuning spectrum you should sit: how much labelled data do you have for the target task, and how similar is your target domain to the source domain the pretrained network saw. Thinking of these as two axes gives four rough regions.

Small dataset, similar domain

Example: a few hundred photographs of five flower species, using a network pretrained on ImageNet (which already contains plenty of plant and flower images among its 1000 classes). Here, plain feature extraction is usually the right call. The pretrained features are already well suited to your kind of image, so you mostly need to learn how to combine them into your specific classes. With so little data, unfreezing the backbone invites overfitting, since you would be adjusting millions of parameters using only a few hundred examples.

Small dataset, different domain

Example: a few hundred chest X-ray images, using the same ImageNet-pretrained network. The domain gap is larger here; X-rays are grayscale, high-contrast, and structurally different from everyday photographs. Feature extraction with a frozen backbone can still help a surprising amount, because low-level edge and texture detectors remain useful, but you may get a further boost by unfreezing only the very last convolutional block and fine-tuning it with a small learning rate, letting the network adjust its highest-level features slightly toward X-ray statistics. Unfreezing much more than that, with only a few hundred images, risks overfitting badly.

Large dataset, similar domain

Example: tens of thousands of product photographs for a retail catalogue classifier. With this much data and a domain close to natural photographs, you can afford to fine-tune a larger portion of the network, sometimes all of it, because you now have enough examples to adjust the deeper layers without memorising noise. Fine-tuning the whole network from a pretrained starting point here typically outperforms pure feature extraction, and it is usually faster and more stable than training from scratch would be.

Large dataset, different domain

Example: tens of thousands of satellite images for land-use classification. The domain gap is large, but you have enough data to let the network relearn a substantial part of its representation. Full fine-tuning of the entire network, starting from pretrained weights rather than random ones, is usually the better choice here. The pretrained weights still provide a head start (the network does not need to rediscover that edges and textures exist), even though many of the higher-level features will end up changing considerably during fine-tuning.

These four regions are a starting intuition, not a rigid rule. In practice it is common to try feature extraction first, since it is fast and cheap, then try a bit of fine-tuning on top of it, and compare the two on a held-out validation set before committing to one strategy.

Building a transfer learning model: replacing the head

Recall from part 4 that a modern CNN classifier like ResNet ends with a global average pooling layer that collapses each feature map down to a single number, followed by one fully connected layer that maps those pooled features to class scores. For an ImageNet-pretrained ResNet, that final layer has 1000 outputs, one per ImageNet class. Since your target task almost certainly does not have 1000 classes, the very first step in transfer learning is to remove that final layer and replace it with a new one sized for your problem.

The following example uses torchvision (version 0.13 or later, where pretrained weights are requested through a weights enum rather than a plain pretrained=True flag) to load a ResNet-18 pretrained on ImageNet and adapt it for a 5-class flower classification problem.

import torch
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights

# Load a ResNet-18 with ImageNet-pretrained weights
weights = ResNet18_Weights.IMAGENET1K_V1
model = resnet18(weights=weights)

# The pretrained model ends with: model.fc = Linear(in_features=512, out_features=1000)
# Replace it with a new head sized for 5 flower classes
num_classes = 5
in_features = model.fc.in_features  # 512 for ResNet-18
model.fc = nn.Linear(in_features, num_classes)

print(model.fc)
# Linear(in_features=512, out_features=5, bias=True)

Notice that only the shape of the final layer changed; everything before it, all the convolutional blocks that compute the 512-dimensional feature vector, is untouched and still holds the weights learned on ImageNet. The new fc layer is initialised randomly, so right after this replacement the model's predictions on your flower images will be close to random guessing. The useful part of the network, the feature extractor, is already competent; only the final decision layer needs to be trained.

Freezing and unfreezing: what it means mechanically

Freezing a layer means telling the training framework not to compute gradient updates for its parameters, so they stay exactly as they were after pretraining. In PyTorch, every parameter tensor has a requires_grad flag; setting it to False excludes that parameter from gradient computation and from any optimiser step that includes it.

import torch
from torchvision.models import resnet18, ResNet18_Weights
import torch.nn as nn

weights = ResNet18_Weights.IMAGENET1K_V1
model = resnet18(weights=weights)

# Freeze every parameter in the pretrained backbone
for param in model.parameters():
    param.requires_grad = False

# Replace the head; new parameters default to requires_grad=True
model.fc = nn.Linear(model.fc.in_features, 5)

# Only the head's parameters will receive gradient updates
trainable_params = [p for p in model.parameters() if p.requires_grad]
print(sum(p.numel() for p in trainable_params))  # should equal 512*5 + 5 = 2565

optimizer = torch.optim.Adam(trainable_params, lr=1e-3)

During the forward pass, data still flows through every frozen layer exactly as before; freezing does not remove or skip layers, it only prevents their weights from changing. During the backward pass, gradients are still computed through the frozen layers as needed to reach earlier frozen layers mathematically, but the optimiser is only given the head's parameters, so only those get updated. This is why feature extraction is relatively fast and memory-light compared to full fine-tuning: fewer parameters need their gradients stored and their optimiser state tracked, even though the forward computation cost is almost the same.

Batch normalization layers need special care

Part 4 introduced batch normalization as a technique that stabilises training by normalising activations using statistics computed from the current batch, while also maintaining running estimates of the mean and variance for use at inference time. These running statistics are not ordinary learnable weights updated by the optimiser; they are updated directly during the forward pass whenever the layer is in training mode, based on whatever batch happens to pass through.

This creates a subtle trap. Setting requires_grad=False on a batch normalization layer's scale and shift parameters stops those two parameters from changing, but if the layer is still in training mode (model.train()), its running mean and variance will keep updating using statistics from your target dataset, which may have very different batch sizes or image statistics than the original pretraining data. This can quietly shift the frozen backbone's behaviour away from what it learned, even though you intended to freeze it completely. The common fix is to explicitly put frozen batch normalization layers into evaluation mode so their running statistics stop updating as well.

def freeze_backbone(model):
    for param in model.parameters():
        param.requires_grad = False
    # Put every module into eval mode so BatchNorm running stats stop updating
    model.eval()

# Later, when training, re-enable train mode only for the new head
freeze_backbone(model)
model.fc.train()  # the new classifier head should still behave as 'training'

If instead you plan to fine-tune the backbone (not just the head), you generally want batch normalization layers to behave normally during training, updating their statistics to match your target data, which is one more reason fine-tuning tends to work better than feature extraction once you have enough data to support it safely.

Fine-tuning strategy: smaller learning rates and gradual unfreezing

When you decide to unfreeze part or all of the backbone, the learning rate becomes critical. The pretrained weights already encode useful structure; a learning rate large enough to be appropriate for a randomly initialised head can be destructive for pretrained weights, because a few large gradient steps can wash out the fine-grained structure those weights encode. This phenomenon, where fine-tuning erases useful pretrained knowledge faster than it adapts it, is sometimes called catastrophic forgetting. The usual remedy is to use a learning rate for the backbone that is noticeably smaller than the one used for the newly initialised head, often by a factor of ten or more.

A convenient way to apply different learning rates to different parts of a network in PyTorch is to pass parameter groups to the optimiser, each with its own learning rate.

import torch
from torchvision.models import resnet18, ResNet18_Weights
import torch.nn as nn

weights = ResNet18_Weights.IMAGENET1K_V1
model = resnet18(weights=weights)
model.fc = nn.Linear(model.fc.in_features, 5)

# Separate the backbone parameters from the new head's parameters
head_params = list(model.fc.parameters())
head_param_ids = {id(p) for p in head_params}
backbone_params = [p for p in model.parameters() if id(p) not in head_param_ids]

optimizer = torch.optim.Adam([
    {"params": backbone_params, "lr": 1e-5},  # small LR: adjust gently
    {"params": head_params, "lr": 1e-3},       # larger LR: learn from scratch
])

A related and often effective technique is gradual unfreezing. Rather than unfreezing the whole backbone on day one, you train with everything frozen except the head for a few epochs, let the head settle into reasonable values, then unfreeze the last convolutional block and continue training with a small learning rate, then perhaps unfreeze one more block after that if your validation performance keeps improving and you have enough data to support it. The idea is that the head should not be fed through a backbone that is simultaneously being perturbed; stabilise the head first, then slowly allow deeper layers to adjust. This mirrors a broader pattern you have likely noticed throughout this path: build from something that works, then relax constraints carefully rather than all at once.

Preprocessing must match what the network expects

Part 1 established that an image fed into a network is a tensor of pixel values, and that how you scale and normalise those values matters. Every pretrained model was trained with a specific preprocessing pipeline, and feeding it images prepared differently will degrade performance without raising any error, which makes this an easy and silent mistake to make.

For models pretrained on ImageNet through torchvision, this typically means resizing images so the shorter side is a certain size (224 pixels is common for many architectures), converting pixel values to the 0 to 1 range, and then normalising each colour channel using the specific mean and standard deviation values computed over the original ImageNet training set, commonly quoted as mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] for the red, green and blue channels respectively. Libraries that distribute pretrained weights, such as torchvision's weights enums, usually expose the exact transform the model expects, which is the safest way to get this right rather than hand-copying numbers.

from torchvision.models import ResNet18_Weights

weights = ResNet18_Weights.IMAGENET1K_V1
preprocess = weights.transforms()  # returns the exact preprocessing pipeline used in training

print(preprocess)
# Something like: resize to 256, center crop to 224, convert to tensor,
# normalize with ImageNet mean and std

Mismatches to watch for include: feeding images in the 0 to 255 range when the model expects 0 to 1 (or vice versa), feeding BGR-ordered channels to a model trained on RGB images (common when mixing OpenCV, which loads images in BGR, with a model that expects RGB), using a very different input resolution than the model was trained on, and skipping channel normalisation entirely. None of these mistakes will usually crash your code; they just quietly produce a confused network with lower accuracy than a correctly preprocessed pipeline would reach, which is exactly the kind of bug that is hard to notice without checking carefully.

A worked example: adapting a flower classifier

To make the strategy concrete, consider classifying photographs into five flower species, with roughly 150 labelled photographs per class, giving about 750 images in total, split into training and validation sets. This sits in the small dataset, similar domain quadrant of the decision matrix from earlier, since ImageNet already contains many plant and flower images among its 1000 categories.

A reasonable first approach is pure feature extraction: load a pretrained ResNet, freeze the entire backbone, replace the final layer with a 5-output linear layer, and train only that layer using the augmentation techniques from part 5 (random crops, flips, modest colour jitter) to squeeze more variety out of the limited data. Because only a few thousand parameters are being trained, this converges quickly, typically within a small number of epochs, and is unlikely to overfit badly even with only 750 images, since the frozen backbone cannot memorise your specific training photographs; it can only produce the same fixed feature representation regardless of how long you train.

If validation accuracy from feature extraction looks promising but leaves room for improvement, a natural next step is light fine-tuning: unfreeze the last residual block of the backbone (not the whole network), set its learning rate to something like a tenth or a hundredth of the head's learning rate, and continue training for a further set of epochs while watching the validation curve closely. If validation performance keeps improving, the extra flexibility is helping; if validation performance starts to diverge from training performance (training accuracy keeps rising while validation accuracy stalls or drops), that is the signature of overfitting, and it is a sign to stop unfreezing further, reduce the learning rate more, or add stronger augmentation. Because full hands-on training of exactly this kind of model is the subject of part 12 in this path, this article stops at the level of strategy and mechanics rather than reporting a final training run; the point here is to understand why each choice is made, so that when you do train the real thing, the decisions are not arbitrary.

Common mistakes

• Mismatched preprocessing: using different normalisation statistics, pixel ranges or channel ordering than the pretrained model expects, which silently lowers accuracy without any error message.
• Learning rate too high during fine-tuning: large gradient steps on pretrained weights can erase useful structure faster than they adapt it, a failure sometimes called catastrophic forgetting. Backbone learning rates should usually be much smaller than head learning rates.
• Unfreezing everything immediately on a tiny dataset: with only a few hundred images, updating tens of millions of backbone parameters invites severe overfitting. Start with feature extraction or partial unfreezing and expand only if validation metrics support it.
• Forgetting batch normalization statistics: freezing a layer's learnable weights without also putting it into evaluation mode allows its running mean and variance to keep drifting based on your target data's batches, quietly undermining the intended freeze.
• Comparing frozen and fine-tuned models unfairly: giving one strategy many more training epochs, a different learning rate schedule, or different augmentation than the other, then concluding one approach is inherently better, when the comparison was never controlled.
• Ignoring input resolution mismatches: feeding a network trained on 224 by 224 images with much smaller source images stretched up to that size does not add real detail; the network still only has access to the information the original image contained.
• Applying augmentation so aggressive that it changes the images far more than what the network saw during pretraining, which can make the fine-tuning phase harder rather than easier (see part 5 for how to tune augmentation strength sensibly).
• Data leakage between training and validation sets, for instance including near-duplicate photographs or multiple frames from the same video in both splits, which inflates validation accuracy and hides real generalisation problems, regardless of which transfer strategy you use.

When domains are very different: diminishing returns and domain adaptation

Transfer learning is not magic; its benefit shrinks as the gap between source and target domains grows. A network pretrained on everyday colour photographs still tends to help with grayscale medical images, satellite imagery or microscope slides, mostly because its early layers encode genuinely general-purpose edge and texture detectors, but the improvement over training from scratch is usually smaller than it would be for another everyday-photograph task, and more of the network typically needs to be fine-tuned, not just the head, to recover good performance.

In rare cases, when the source and target domains are extremely dissimilar or the source task's inductive biases actively conflict with the target task's requirements, transfer can even slightly hurt compared to training from scratch, a phenomenon sometimes called negative transfer. This is uncommon in everyday computer vision work but worth knowing about so you do not assume transfer learning is guaranteed to help in every situation; it is a strong default, not an unconditional law. When domain gaps are large and labelled target data is scarce, a related and more specialised set of techniques called domain adaptation exists specifically to close that gap, though that is a broader topic than this article covers.

Where this fits in the path

Transfer learning is less a standalone technique than a foundation that much of modern computer vision is built on top of. The object detectors and segmentation models you will meet in parts 7 and 8 almost always start from a pretrained backbone, exactly like the one you adapted here, with detection or segmentation heads added on top rather than a simple classification head. The vision transformer architectures in part 9 are typically also distributed as pretrained models and adapted to new tasks using the same feature-extraction-to-fine-tuning spectrum described in this article, even though their internal structure differs from the convolutional networks covered in parts 2 through 4. Part 10 introduces self-supervised pretraining, an alternative way of producing a useful pretrained backbone without needing millions of human-provided labels in the first place, which you can then transfer and fine-tune using exactly the techniques covered here. Part 12 puts all of this into practice as a full hands-on project, fine-tuning a ResNet on a custom photo dataset from loading the data through to evaluating the final model.

Summary and what comes next

Training a deep CNN from scratch requires far more labelled data and compute than most real projects have available. Transfer learning avoids this by reusing a network already trained on a large dataset, on the premise that its early and middle layers have learned broadly useful visual features, such as edges, textures and object parts, that are not tied to the specific classes it was originally trained on. Feature extraction freezes this pretrained backbone and trains only a new head, which works well with small datasets in domains similar to the pretraining data. Fine-tuning goes further, continuing to update some or all of the backbone's weights, usually with a smaller learning rate than the head and sometimes through gradual unfreezing, and tends to work better once you have more target data or a larger domain gap to bridge. Getting the details right matters: matching the original preprocessing, handling batch normalization carefully when freezing, and choosing learning rates that adapt pretrained weights gently rather than erasing them. The next article turns to object detection, where you will see these same pretrained backbones repurposed not just to classify an entire image, but to locate and label multiple objects within it.