The problem self-supervised learning solves

Every model you have built so far in this path has needed labels. In Part 6 you fine-tuned a pretrained network on a labeled dataset. In Part 7 and Part 8 the detectors and segmenters needed boxes or masks drawn by hand. Labels are the fuel, and they are expensive: someone has to look at each image and decide what is in it, where it is, or which pixels belong to it. For a dataset of a few thousand images this is a weekend of work. For the billions of images on the internet it is simply not possible.

Self-supervised learning is a way to get useful representations from images without any human labels at all. The trick is to invent a task that can be scored automatically from the image itself, called a pretext task, and train a network to solve it. Solving the pretext task forces the network to learn something about the structure of images in general: edges, textures, object parts, shapes, how colour and lighting vary, how parts relate to wholes. Once the network has learned this, you can reuse its weights the same way you reused ImageNet weights in Part 6, except now the pretraining did not need a single labeled example.

This article covers the two families of pretext tasks that currently dominate self-supervised vision: contrastive learning, which teaches a network to recognise when two views come from the same image, and masked image modeling, which teaches a network to reconstruct parts of an image it cannot see. Both were covered only in passing in earlier parts; here we go through the mechanics in enough detail that you could implement a small version yourself.

Early pretext tasks, and why they were replaced

Before contrastive and masked methods became standard, researchers tried simpler pretext tasks. A few examples give useful intuition for what makes a good pretext task.

  • Predict the rotation applied to an image (0, 90, 180 or 270 degrees). To get this right, the network has to understand what objects normally look like upright, which requires some notion of shape and orientation.
  • Predict the relative position of two patches cut from the same image (is patch B above, below, or beside patch A). This forces the network to understand spatial layout and object parts.
  • Colourize a greyscale image. This forces the network to associate textures and object identity with plausible colours, for example learning that sky tends to be blue and leaves tend to be green.

These tasks worked, in the sense that representations learned this way transferred better than random initialization to downstream tasks like classification. But the transfer quality was noticeably below representations learned from labeled ImageNet pretraining, which was the benchmark at the time. The pretext task and the downstream task were too loosely connected: a network can get very good at predicting rotation angle using shortcuts, such as looking at compression artefacts or borders, without learning much about object content. Contrastive and masked methods close this gap, and on some benchmarks and some backbones they now match or exceed supervised pretraining. The rest of this article explains why they work better.

Contrastive learning: same image, different views

The core idea

Contrastive learning starts from a simple assumption: if you take one image and apply two different random augmentations to it, crop, colour jitter, flip, blur, the two results should still be recognisable as the same underlying scene to a good representation, even though the pixels are quite different. Recall from Part 5 that augmentations like random crop and colour jitter are designed to change the image without changing its label. Contrastive learning turns that idea into a training signal: pull the representations of two augmented views of the same image close together, and push the representations of views from different images apart.

Concretely, take a batch of images. For each image, generate two augmented views, so a batch of N images becomes 2N views. The two views from the same original image form a positive pair. Every other view in the batch is treated as a negative for that pair. A network, usually a CNN like a ResNet (Part 4) or a Vision Transformer (Part 9), maps each view to a vector, called an embedding. A loss function then rewards the network for making the embeddings of a positive pair similar to each other and dissimilar from the embeddings of negatives.

SimCLR and the NT-Xent loss

One of the clearest frameworks for contrastive learning is SimCLR. Its pipeline has four pieces:

  1. A strong augmentation pipeline that produces two views per image: random resized crop, random horizontal flip, colour jitter, and occasionally greyscale conversion or Gaussian blur.
  2. An encoder network, for example a ResNet-50, that maps an image view to a feature vector, typically a few thousand numbers.
  3. A small projection head, usually one or two fully connected layers, that maps the encoder's feature vector down to a smaller embedding space, for example 128 numbers, where the contrastive loss is actually computed.
  4. A contrastive loss called NT-Xent, short for normalized temperature-scaled cross entropy, applied to the embeddings.

The projection head is a detail worth remembering: after pretraining, it is thrown away, and only the encoder is kept for downstream tasks. The reasoning is that the projection head learns to discard information that is useful for the contrastive task but not necessarily useful for other tasks, for instance it might learn to ignore colour because colour jitter was applied, while the encoder beneath it still preserves that information for later use.

The NT-Xent loss works like this. First, every embedding is normalized to unit length, so similarity between two embeddings can be measured with a simple dot product, which is then exactly the cosine similarity. For an anchor view, its positive is the other view of the same image, and all other 2N minus 2 views in the batch are negatives. The loss for that anchor is the cross-entropy of a softmax over similarities to all other views (positive and negatives), scaled by a temperature parameter, with the positive playing the role of the correct class.

In symbols, if sim(u, v) is the cosine similarity between embeddings u and v, and tau is the temperature, the loss for anchor i with positive j is: L(i, j) = -log( exp(sim(i, j) / tau) / sum over all k not equal to i of exp(sim(i, k) / tau) ) The total loss averages this over every anchor in the batch, using each view in turn as the anchor.

A worked example with four views

Numbers make this concrete. Suppose we have a tiny batch of two images, A and B, each with two augmented views, giving four embeddings. To keep the arithmetic simple we work in two dimensions and place each embedding on the unit circle (unit length, so dot product equals cosine similarity). Say: A1 is at angle 0 degrees: (1.000, 0.000) A2 is at angle 30 degrees: (0.866, 0.500) B1 is at angle 90 degrees: (0.000, 1.000) B2 is at angle 100 degrees: (-0.174, 0.985) A1 and A2 are the two views of image A, close together because they came from the same photo. B1 and B2 are the two views of image B, also close together. A's views and B's views are far apart on the circle, as they should be since they come from different images.

Using a temperature of tau equals 0.5, take A1 as the anchor. Its positive is A2, with cosine similarity 0.866, giving exp(0.866 / 0.5) = exp(1.732) is about 5.65. The negatives are B1 (similarity 0.000, exp(0) = 1.00) and B2 (similarity -0.174, exp(-0.347) is about 0.71). The denominator sums the positive and both negatives: 5.65 + 1.00 + 0.71 is about 7.36. The loss for this anchor is -log(5.65 / 7.36), which is -log(0.768), about 0.264.

Doing the same calculation with A2, B1 and B2 each as the anchor in turn (each time using the other member of its pair as the positive and the two views of the other image as negatives) gives losses of roughly 0.605, 0.418 and 0.318. Averaging all four gives a batch loss of about 0.40. The code below reproduces this exactly, so you can check the arithmetic and experiment with different temperatures or embedding positions.

🐍Python
import numpy as np

# Four unit-length embeddings: two views of image A, two views of image B
angles_deg = {"A1": 0, "A2": 30, "B1": 90, "B2": 100}
names = list(angles_deg.keys())
embeddings = np.array([
    [np.cos(np.radians(a)), np.sin(np.radians(a))] for a in angles_deg.values()
])

tau = 0.5
# Index 0,1 -> image A views; index 2,3 -> image B views
positive_of = {0: 1, 1: 0, 2: 3, 3: 2}

sim = embeddings @ embeddings.T  # cosine similarity since vectors are unit length
exp_sim = np.exp(sim / tau)
np.fill_diagonal(exp_sim, 0.0)   # an anchor is never its own negative or positive

losses = []
for i in range(4):
    j = positive_of[i]
    numerator = exp_sim[i, j]
    denominator = exp_sim[i].sum()
    loss_i = -np.log(numerator / denominator)
    losses.append(loss_i)
    print(f"anchor {names[i]}, positive {names[j]}: loss = {loss_i:.3f}")

print(f"batch NT-Xent loss: {np.mean(losses):.3f}")

Running this prints a loss close to 0.264, 0.605, 0.418 and 0.318 for the four anchors, averaging to about 0.40, matching the hand calculation above (small differences of a thousandth or two can appear from rounding the angles). Two things are worth noticing. First, the loss is low when positives are close and negatives are far, which is exactly the behaviour we want to encourage during training; gradient descent will nudge the embeddings further in this direction over many batches of real images. Second, the loss depends on every other view in the batch, not just the positive pair, because the denominator sums over all of them. This is why contrastive learning historically needed very large batches: more negatives per positive pair gives the network a harder, more informative task, and a sharper training signal.

Why large batches (or memory banks) matter

In the toy example there were only two negatives. In practice, SimCLR used batch sizes in the thousands, so each positive pair is contrasted against thousands of negatives drawn from other images in the batch. A small number of negatives makes the task too easy: distinguishing a photo of a dog from one random unrelated photo is a weak signal. Distinguishing it from a thousand different photos, some of which may contain other dogs, forces the representation to capture fine-grained, genuinely useful detail. The downside is that very large batches need a lot of accelerator memory, which was a practical barrier to training SimCLR-style models.

A method called MoCo (momentum contrast) addressed this without needing enormous batches. Instead of drawing negatives only from the current batch, MoCo keeps a queue of embeddings computed in previous batches, so the effective pool of negatives can be large (tens of thousands) while the actual batch size stays modest. To keep the queued embeddings consistent with the current encoder even though they were computed by slightly older weights, MoCo uses two encoders: a regular one updated by gradient descent, and a momentum encoder whose weights are an exponential moving average of the first. The momentum encoder changes slowly and smoothly, so embeddings it produced a few steps ago are still a fair comparison for embeddings produced now.

Avoiding collapse without negatives: BYOL and SimSiam

An obvious question is whether negatives are even necessary. If you only pull positive pairs together and never push anything apart, what stops the network from mapping every image to the exact same embedding, which would make the positive-pair loss zero while learning nothing useful? This failure mode is called representation collapse.

BYOL (bootstrap your own latent) and SimSiam showed that collapse can be avoided without explicit negatives, through architectural asymmetry rather than a contrastive term in the loss. Both use two networks processing the two augmented views: an online network that is trained normally, and a target network whose role is only to provide a training signal. In BYOL the target network is a momentum-averaged copy of the online network, similar in spirit to MoCo's momentum encoder, but used here to prevent collapse rather than to extend a negative queue. The online network is also given an extra small prediction head on top of its projection, and the loss asks the online network's prediction to match the target network's projection, with gradients only flowing through the online side. This asymmetry, a slowly changing target plus a one-sided predictor, turns out to be enough to stop the trivial constant-output solution from being reachable by gradient descent, even though nothing in the loss explicitly forbids it. SimSiam showed that even the momentum-averaged target can be replaced by a simple stop-gradient on a shared-weight twin network, making the recipe simpler still.

The practical takeaway is that there are now two broad contrastive-style recipes: ones that need explicit negatives and therefore large batches or memory queues (SimCLR, MoCo), and ones that avoid negatives through careful architectural asymmetry (BYOL, SimSiam). Both families have produced strong pretrained encoders; which one is easier to use well depends on your compute budget, since avoiding the negative-sampling machinery simplifies implementation but still requires getting the momentum and stop-gradient details right.

Masked image modeling: fill in the missing pieces

Borrowing an idea from language models

The second major family of self-supervised methods takes a different pretext task: hide part of the image and train the network to reconstruct what was hidden. This mirrors masked language modeling in text, where a model learns by predicting words that have been blanked out of a sentence. Masked image modeling became practical for vision once Vision Transformers (Part 9) made it natural to think of an image as a sequence of patches, because a transformer can simply drop some patch tokens from its input sequence the way a language model drops word tokens.

How a Masked Autoencoder (MAE) works

The best-known version of this idea is the Masked Autoencoder, or MAE. Recall from Part 9 that a Vision Transformer splits an image into a grid of fixed-size patches, for example 16 by 16 pixels, flattens each patch, and feeds the sequence of patch embeddings through transformer blocks. MAE adapts this pipeline for self-supervised pretraining as follows.

  1. Split the image into patches, as in a standard ViT, for example a 224 by 224 image split into 16 by 16 patches gives a 14 by 14 grid, or 196 patches.
  2. Randomly mask a large fraction of the patches, commonly around three quarters of them, and simply remove them from the sequence rather than replacing them with a special mask token at this stage.
  3. Feed only the small set of visible patches, roughly a quarter of the full grid, through the encoder, which is a regular ViT. Because most patches are removed before the encoder even sees them, the encoder's work is cheap: it processes a short sequence.
  4. Add placeholder mask tokens back in for every position that was removed, combine them with the encoded visible patches plus positional information, and pass the full-length sequence through a lightweight decoder, typically a much smaller transformer than the encoder.
  5. Train the decoder to predict the original pixel values of the masked patches, using a simple loss such as mean squared error between predicted and actual pixel values, computed only on the masked positions.

Two design choices stand out and are worth understanding rather than memorising, because they explain why MAE is efficient. The first is masking a very high fraction of patches, far higher than the roughly 15 percent typically masked in language models. Images have a lot of redundancy between neighbouring pixels and patches; a model could cheat by just interpolating from adjacent unmasked patches if too few were hidden, so the masking ratio has to be aggressive enough that reconstruction genuinely requires understanding the scene, for example inferring that a mostly-hidden shape is a car because of the visible wheel and bumper. The second is removing masked patches from the encoder's input entirely rather than feeding mask tokens through it. Since only a quarter of patches reach the encoder, the heavy part of the network runs on a short sequence, which is where most of the pretraining compute saving comes from. Mask tokens are only introduced at the lightweight decoder stage, where reconstructing pixels is a comparatively easy and cheap task.

A small worked example of the reconstruction target

To make the loss concrete, imagine a toy image of only 4 patches arranged in a 2 by 2 grid, each patch a single greyscale value for simplicity: patch values are 10, 200, 190, 20 (reading left to right, top to bottom; imagine this as a tiny checkerboard-like image, dark, light, light, dark). Suppose the mask hides two of the four patches, say the top-right (200) and bottom-left (190), leaving the encoder to see only the dark corners, 10 and 20. The decoder, seeing the two visible values plus mask tokens at the two hidden positions plus positional information telling it where each token belongs in the grid, outputs some predictions, say 180 and 170 for the two masked positions. The reconstruction loss is the mean squared error on just those two positions: ((200-180) squared + (190-170) squared) divided by 2, which is (400 + 400) / 2 = 400. Training adjusts the network's weights, over many such examples, to shrink this error, which only happens if the network has learned something general about how patch values relate to their neighbours and position, since it never got to see the true values of the masked patches directly.

Real MAE patches are 16 by 16 pixels with three colour channels, so each patch is a vector of 768 numbers rather than a single scalar, and the loss is averaged over all those numbers and over all masked patches, but the mechanism is exactly the toy example scaled up: predict the hidden numbers from the visible ones plus position, and get penalised by squared error for being wrong.

Contrastive versus masked: how they compare

Both families produce encoders that transfer well to downstream tasks, but they encode somewhat different biases, and it is worth understanding the practical differences rather than treating them as interchangeable.

  • Backbone fit: masked image modeling as used in MAE was designed around the Vision Transformer's patch-sequence structure, specifically the ability to drop tokens before the encoder. Contrastive methods are more backbone-agnostic and were originally developed and tuned with convolutional networks like ResNet-50, though they also work with transformers.
  • Augmentation dependence: contrastive learning's whole signal comes from the augmentation pipeline, since the positive pair is defined entirely by which augmentations were applied to the same source image. This means the choice of augmentations (crop scale, colour jitter strength, whether to include blur) strongly shapes what the representation learns to be invariant to, and getting this pipeline wrong noticeably hurts downstream performance. MAE depends far less on a hand-designed augmentation r
  • Batch size and infrastructure: methods relying on explicit negative pairs (SimCLR, MoCo) are more sensitive to batch size or need extra machinery like a memory queue. MAE's masking strategy does not have this requirement in the same way, since the training signal does not come from comparing different images against each other.
  • What gets evaluated: contrastive encoders are usually evaluated by how well a single linear layer on top of their frozen features can classify images, since the embedding space is explicitly trained to make similar things close and different things far. Masked autoencoders are more often evaluated after fine-tuning the whole encoder, since the representation is optimised for pixel reconstruction, which is a different objective from linear separability, and fine-tuning lets the network adapt its

Neither family strictly dominates the other across every setting, backbone and dataset size, and the relative ranking has shifted as both approaches have matured, so it is best to think of them as two different, complementary tools in a self-supervised toolbox rather than a settled competition with one clear winner.

Using a self-supervised encoder in practice

Pretraining with a contrastive or masked objective is only step one. The reason to go through it is to get a good starting point for a real task where labels are scarce, which connects directly back to the transfer learning ideas from Part 6. There are two standard ways to use the resulting encoder.

Linear probing

Freeze every weight in the pretrained encoder, run your labeled images through it to get fixed feature vectors, and train only a new linear classifier (a single fully connected layer) on top, using your labeled dataset. This is a fast, cheap way to measure how good the representation is, since the result depends almost entirely on the quality of the frozen features rather than on how carefully you tuned a fine-tuning schedule. It is the standard way self-supervised methods are compared against each other in research, precisely because it isolates representation quality from fine-tuning tricks.

Fine-tuning

Initialise a network with the pretrained encoder's weights, add a task-specific head (a classifier, a detection head as in Part 7, or a segmentation head as in Part 8), and continue training all or most of the weights on your labeled data, usually with a small learning rate as discussed in Part 6. Fine-tuning generally gets better final accuracy than a frozen linear probe, because the features can adjust to the specifics of the downstream task, but it needs more labeled data and compute than linear probing, and it is slower to iterate on since you cannot cache the frozen features.

A practical workflow for a project with limited labels, for example the plant disease classifier you will build in Part 14, looks like this: start from an encoder pretrained with a self-supervised method (or, just as reasonably, start from a supervised ImageNet checkpoint as in Part 6, since self-supervised pretraining is an alternative to that, not a replacement for the whole transfer learning idea), run a quick linear probe to sanity-check that the features separate your classes reasonably well, and then fine-tune the whole network on your labeled set for the final model.

Common mistakes and things worth checking

  • Treating augmentation choices as an afterthought in contrastive learning. Since the positive pair is defined by the augmentations, weak or mismatched augmentations (for example, no colour jitter when your downstream task genuinely needs colour-invariant features, or overly aggressive cropping that removes the object entirely) directly degrade what the encoder learns. Review Part 5 before designing a contrastive pipeline.
  • Evaluating only with a linear probe and assuming the encoder will fine-tune just as well, or vice versa. Linear probe accuracy and fine-tuned accuracy do not always rank methods the same way, so if your end goal is fine-tuning, it is worth checking fine-tuned performance rather than relying solely on probe numbers from a paper.
  • Using a batch size far smaller than what a contrastive method was designed for, then being surprised that results are weak. If you cannot afford a large batch, consider a method built for that constraint, such as MoCo's queue-based negatives or a non-contrastive method like BYOL or SimSiam, rather than running SimCLR with a tiny batch.
  • Assuming masking ratio does not matter for masked image modeling. Masking too little makes the reconstruction task too easy to be useful for representation learning; masking too much leaves too little information for the task to be learnable at all. The high masking ratios reported for MAE were chosen deliberately after experimentation, not as an arbitrary default.
  • Forgetting to drop the projection head after contrastive pretraining. The projection head is trained to serve the narrow goal of the contrastive loss and typically discards information that is useful more broadly; keeping it attached for downstream tasks usually performs worse than using only the encoder's output.
  • Comparing a self-supervised encoder pretrained on a small, narrow dataset against one pretrained on a large and diverse one, and attributing the gap entirely to the method. Self-supervised learning still benefits strongly from pretraining data scale and diversity, in a similar way to supervised pretraining; a fair comparison needs matched pretraining data.

Where this matters beyond benchmarks

The practical appeal of self-supervised pretraining is strongest exactly where labels are hardest to get: medical imaging, where expert annotation time is scarce and expensive; satellite and aerial imagery, where there is an enormous volume of unlabeled data but comparatively few labeled examples of rare events; industrial inspection, where a factory may have huge archives of unlabeled images of parts but few examples of specific defects labeled by an engineer; and robotics, where a robot's camera produces a continuous stream of images that are far easier to collect than to label. In all of these settings, the recipe is the same one described above: pretrain an encoder with a contrastive or masked objective on the large pool of unlabeled images from the relevant domain, then fine-tune or linear-probe on whatever smaller labeled set exists for the actual task. Pretraining on images from the target domain, rather than generic internet photos, tends to help further, since the encoder's reconstruction or contrastive task gets to learn the specific textures, structures and layouts relevant to that domain.

Summary and what comes next

Self-supervised learning replaces human labels with a pretext task that can be scored automatically from the image alone. Contrastive learning, as in SimCLR, MoCo, BYOL and SimSiam, trains an encoder so that different augmented views of the same image end up with similar embeddings while views of different images end up with dissimilar ones, with the main design challenge being how to supply useful negative examples, or how to avoid collapse when you deliberately do not. Masked image modeling, as in MAE, trains an encoder and a lightweight decoder to reconstruct patches of an image that were deliberately hidden, taking advantage of the Vision Transformer's patch-sequence structure to make the heavy encoder cheap to run. Both produce encoders that transfer well to downstream tasks through linear probing or fine-tuning, following the same transfer learning logic introduced in Part 6, just with a pretraining phase that no longer depends on labeled data.

With this part, the path has now covered how CNNs and Vision Transformers are built, trained and pretrained. Part 11 turns to a different question: once you have a trained model, how do you find out what it is actually looking at in an image, using tools like Grad-CAM and saliency maps. Parts 12 through 14 then move to hands-on work, applying everything from this path, including the transfer learning and pretraining ideas from this article, to fine-tune a ResNet on your own photos, train a YOLO detector on a custom dataset, and build a complete plant disease classifier served as an API.