Vision Transformers

From stacking filters to attending to patches

Every architecture in this path so far, from LeNet to ResNet, from YOLO's anchors to U-Net style segmentation heads, has been built on the same basic operation: convolution. A small filter slides over the image, looks at a local neighbourhood of pixels, and produces one number per position. Stack enough of these layers and the network gradually builds up a view of the whole image, starting from edges and textures and ending with shapes and objects.

Vision Transformers, usually shortened to ViT, take a different route. Instead of sliding a small filter over the image, they cut the image into a grid of patches, turn each patch into a vector, and let every patch look directly at every other patch through a mechanism called self-attention. There is no sliding window and no built-in idea of locality. The network has to learn, from data, which parts of the image are relevant to which other parts. This idea comes from natural language processing, where the original transformer architecture let every word in a sentence attend to every other word. The paper that adapted this to images is usually cited by its memorable title, An Image is Worth 16x16 Words, published by Dosovitskiy and colleagues in 2020.

This article builds the ViT architecture piece by piece: how an image becomes a sequence, how self-attention works with a small worked numeric example, how the pieces are assembled into a full encoder, and then the practical question every student of computer vision eventually asks: when should you reach for a transformer instead of a CNN.

What CNNs give you for free, and what ViTs don't

Part 4 of this path talked about why convolutional networks worked so well for images: they bake in two assumptions about the world, often called inductive biases. The first is locality, the idea that a pixel is mostly related to its near neighbours, so a small filter looking at a small patch is a sensible building block. The second is translation equivariance, the idea that an object looks the same wherever it sits in the image, so the same filter can be reused at every position by sliding it across the image.

These two assumptions are extremely useful because they are usually true for natural images, and they mean a CNN does not have to learn from scratch that a cat in the top-left corner is the same kind of thing as a cat in the bottom-right corner. The network starts out already knowing this, encoded directly into its architecture.

A plain Vision Transformer has neither of these biases built in. Self-attention treats the image as an unordered set of patches that happen to carry position information attached as extra data, not as a 2D grid with built-in neighbourhood structure. This is not a design flaw, it is a deliberate trade-off. Giving up built-in locality means the network must learn spatial relationships from the data itself, which takes more examples, but it also means the network is free to learn relationships that convolution could never express directly, such as a far corner of the image being relevant to understanding an object in the centre. Keep this trade-off in mind, it explains almost everything else in this article: why ViTs need a lot of data, why hybrid and hierarchical designs exist, and why CNNs are still a sound default for smaller datasets.

Turning an image into a sequence of patches

Transformers, as originally designed for text, expect a sequence of vectors, one vector per word or sub-word token. A full-resolution image has no such sequence built in, it is a grid of pixels. ViT solves this with a simple trick: cut the image into a grid of fixed-size square patches, and treat each patch as one token.

Splitting the image into patches

Take a typical input size used in the original ViT paper, a 224 by 224 pixel RGB image, and a patch size of 16 by 16 pixels. Along each dimension, 224 divided by 16 gives 14, so the image is cut into a 14 by 14 grid, which is 196 patches in total. Each patch is a small block of 16 by 16 pixels across 3 colour channels, so as a flat vector it has 16 times 16 times 3, which is 768 numbers. Notice that this patch size has to divide the image size evenly, which is one practical constraint to remember: if you resize your input images, the number of patches changes, and so does the length of the sequence the transformer has to process.

Patch embedding: turning a patch into a vector

A flattened patch of 768 raw pixel values is not yet in a useful representation, it is just pixels rearranged into a list. ViT passes each flattened patch through a single learned linear layer that projects it into an embedding of some chosen dimension, commonly called the hidden size or model dimension and usually written as D. In the base configuration from the original paper, D is 768, the same as the patch's raw pixel count, though this is a coincidence of that particular configuration rather than a requirement; larger ViT variants use a larger D, for instance the paper's large configuration uses a hidden size of 1024 and its huge configuration uses 1280. This single linear projection is the entire patch embedding step: no convolutional feature extractor, no pooling, just one matrix multiplication per patch.

A convenient implementation detail worth knowing, because you will see it in almost every ViT codebase, is that this patch-and-project step can be done in one line using an ordinary convolutional layer whose kernel size and stride both equal the patch size. A kernel that is exactly the size of a patch, moved with a stride equal to its own size so that patches never overlap, computes exactly the same operation as flattening each patch and multiplying it by a weight matrix.

import torch
import torch.nn as nn

# a single random RGB image, batch size 1, 224x224 pixels
image = torch.randn(1, 3, 224, 224)

patch_size = 16
embed_dim = 768

# kernel_size = stride = patch_size turns this convolution into
# "cut into non-overlapping patches, then linearly project each one"
patch_embed = nn.Conv2d(
    in_channels=3,
    out_channels=embed_dim,
    kernel_size=patch_size,
    stride=patch_size,
)

features = patch_embed(image)          # shape: (1, 768, 14, 14)
tokens = features.flatten(2).transpose(1, 2)  # shape: (1, 196, 768)

print(tokens.shape)
# torch.Size([1, 196, 768])

This code assumes a recent version of PyTorch, anything from the 1.x or 2.x series will work the same way since nn.Conv2d and basic tensor reshaping have been stable for a long time. The output shape, 196 by 768, is the sequence the rest of the transformer will work with: 196 tokens, one per patch, each represented by a 768-number vector. From this point on, the transformer encoder does not know or care that these tokens came from an image, they are just a sequence of vectors, exactly the shape it would expect from a sentence of 196 words each embedded in 768 dimensions.

Position embeddings: telling the model where each patch came from

There is a problem with treating the image as a plain set of 196 vectors: self-attention, as you will see shortly, has no inherent sense of order or position. If you shuffled the 196 patch vectors before feeding them in, a pure self-attention layer would produce the same set of outputs, just shuffled the same way. For text this would be a disaster, since word order changes meaning, and for images it is just as much of a problem, since a patch of sky in the top-left means something different from the same patch of sky appearing in the bottom-right relative to everything else.

ViT fixes this by adding a learned position embedding to every patch embedding before the sequence enters the transformer. There is one position embedding vector per grid location, each with the same dimension D as the patch embeddings, and it is simply added element-wise. These position embeddings start out random and are learned during training, just like every other weight in the network. Once trained, the network has effectively learned its own notion of a 2D grid layout, purely from data, even though nothing in the architecture forced that grid structure on it directly.

The class token

A classifier needs a single vector to summarise the whole image before it can produce class scores. CNNs usually do this with global average pooling at the end, squashing the final feature map down to one vector per channel. ViT borrows a different trick from the original text transformer literature: it prepends one extra learnable vector to the sequence, called the class token, often written as [CLS]. So instead of 196 patch tokens, the sequence fed to the encoder has 197 tokens: the class token plus the 196 patches, each with its own position embedding, where the class token gets its own dedicated position embedding as well.

As the sequence passes through the transformer layers, every token, including the class token, gets updated by attending to every other token. Because self-attention lets the class token look at all 196 patches at every layer, by the final layer the class token's vector has, in principle, gathered information from the whole image. That final class token vector is what gets passed to a small classification head, usually just one linear layer, to produce the output class scores. The patches themselves are discarded for a plain classification task, though in detection and segmentation setups, covered in parts 7 and 8, the patch tokens are exactly what gets used instead.

Self-attention, explained with real numbers

Self-attention is the mechanism that lets every token look at every other token and decide how much to borrow from each one. The easiest way to build intuition is to work through a small example by hand, with made-up but concrete numbers, rather than jumping straight to the general formula.

A toy example: three patches

Imagine a tiny image cut into only three patches, and imagine, purely for the sake of a clean example, that each patch's embedding is just a 2-number vector that roughly captures how much sky-like and how much grass-like content it has. Patch 1 is pure sky, patch 2 is pure grass, and patch 3 sits on the horizon, mixing both.

• x1 (sky) = [1, 0]
• x2 (grass) = [0, 1]
• x3 (horizon) = [1, 1]

Self-attention works with three different projections of each token, called the query, the key and the value. In a real transformer these come from three separate learned weight matrices, Wq, Wk and Wv, each multiplying every token's embedding. To keep this example readable by hand, assume all three matrices happen to be the identity matrix, so query, key and value vectors are all equal to the original embedding. This is not realistic for a trained network, where these matrices learn to extract different, useful aspects of each token, but it keeps the arithmetic simple while still showing exactly how the mechanism combines information.

To compute the new, attention-updated representation of patch 3, the horizon patch, self-attention takes its query vector and compares it against the key vector of every token, including itself, using a dot product. The dot product is a similarity score: it tends to be larger when two vectors point in a similar direction.

• score(x3, x1) = x3 . x1 = (1*1) + (1*0) = 1
• score(x3, x2) = x3 . x2 = (1*0) + (1*1) = 1
• score(x3, x3) = x3 . x3 = (1*1) + (1*1) = 2

Before turning these scores into weights, the transformer divides them by the square root of the key dimension, here the dimension is 2, so the scaling factor is the square root of 2, roughly 1.414. This scaling step exists purely for numerical stability: without it, as the embedding dimension grows into the hundreds, dot products can become very large, pushing the next step, softmax, into regions where its gradients are almost zero and learning stalls. Scaling keeps the numbers in a sensible range regardless of dimension.

• scaled score with x1: 1 / 1.414 = 0.707
• scaled score with x2: 1 / 1.414 = 0.707
• scaled score with x3: 2 / 1.414 = 1.414

These scaled scores are then passed through a softmax function, which turns any list of numbers into a set of positive weights that sum to 1, with larger inputs getting disproportionately larger weights. Using the standard exponential-and-normalise softmax formula on 0.707, 0.707 and 1.414 gives weights of roughly 0.248, 0.248 and 0.503. In other words, patch 3 pays about 25 percent attention to patch 1, 25 percent to patch 2, and about 50 percent to itself.

The final step multiplies each token's value vector by its attention weight and adds them all up, producing the new representation of patch 3:

new x3 = 0.248 * [1, 0] + 0.248 * [0, 1] + 0.503 * [1, 1] = [0.751, 0.751]

The result, [0.751, 0.751], is almost perfectly balanced between the sky-like and grass-like dimensions, which matches intuition: the horizon patch, after attention, ends up as a representation that reflects both of its neighbours fairly evenly, while still weighting its own original content the most. This is the whole idea of self-attention in one small, concrete calculation: every token's new representation is a weighted average of every token's value vector, where the weights are computed from how similar that token's query is to every other token's key.

In a trained ViT, the query, key and value projection matrices are not the identity, they are learned so that, for example, a patch belonging to an object's edge learns to produce a query that strongly matches the keys of other edge patches belonging to the same object, even if those patches are far away in the image. This is precisely the long-range relationship that a small convolutional filter, restricted to a local neighbourhood, cannot see directly without many stacked layers.

Multi-head attention

A single attention computation like the one above can only learn one notion of similarity at a time, because there is only one set of Wq, Wk, Wv matrices. Transformers instead run several of these attention computations in parallel, called heads, each with its own independently learned Wq, Wk and Wv, each working on a smaller slice of the full embedding dimension. The original ViT base configuration, for instance, uses 12 attention heads, each operating on 64 dimensions, since 12 times 64 equals the hidden size of 768.

Each head is free to specialise in a different kind of relationship: one head might learn to connect patches that share similar colour, another might learn to connect patches that are spatially close, another might learn to connect patches belonging to the same object regardless of distance. After all heads finish, their outputs are concatenated back together and passed through one more linear layer to mix the information from different heads into a single updated vector per token. This is directly analogous to how a CNN uses many filters per layer rather than just one, each filter free to detect a different pattern.

The transformer encoder block

Multi-head self-attention is only one half of a transformer layer. A full ViT encoder block wraps it with a few more standard ingredients, each borrowed directly from the original text transformer design:

• Layer normalisation applied before the attention step, which rescales each token's vector to have stable statistics, helping training converge reliably.
• The multi-head self-attention step described above.
• A residual connection, where the block's input is added back to the attention output, exactly the same idea that made ResNet trainable at great depth, as covered in part 4.
• A second layer normalisation, followed by a small two-layer fully connected network, often called the MLP block or feed-forward block, applied independently to each token.
• A second residual connection around that MLP block.

A full ViT stacks many of these encoder blocks on top of each other, 12 of them in the base configuration from the original paper, 24 in the large configuration. Every block updates every token's representation a little further, each time letting tokens exchange information with every other token through self-attention, and then process that gathered information individually through the MLP.

The full architecture, start to finish

Putting every piece together, a plain ViT classifier works like this on one input image:

1. Cut the image into fixed-size, non-overlapping patches, for instance 16 by 16 pixels.
2. Flatten each patch and project it linearly to a D-dimensional embedding, often implemented as one convolution with kernel size and stride equal to the patch size.
3. Prepend a learnable class token to the sequence of patch embeddings.
4. Add a learnable position embedding to every token, including the class token, so the model knows where each patch sits in the grid.
5. Pass the resulting sequence through a stack of transformer encoder blocks, each containing multi-head self-attention and an MLP, wrapped in layer normalisation and residual connections.
6. Take the final representation of the class token after the last block.
7. Pass that single vector through a small classification head, typically one linear layer, to produce class scores.

Notice what is entirely absent from this list: no pooling layers, no local filters, no fixed receptive field growing layer by layer the way it does in a CNN. Every token can, in principle, influence every other token starting from the very first layer. That flexibility is the main architectural difference from everything covered in parts 1 through 8 of this path.

Why ViTs are hungry for data

Recall the trade-off from earlier: CNNs come with locality and translation equivariance built in, so they do not need to rediscover these facts about images from scratch. A plain ViT has neither bias wired into its architecture, it only has position embeddings that give it some access to location, and it must learn everything else, including the fact that nearby patches are usually related, purely by seeing enough examples where that pattern helps minimise the loss.

This is exactly what the original ViT paper found and reported clearly: trained from scratch on a moderately sized labelled dataset, a plain ViT tends to underperform a comparable CNN, but its performance improves substantially as the amount of pretraining data grows into the very large regime, eventually matching or surpassing strong CNNs of similar computational cost. The practical lesson is not that transformers are simply better or worse than convolutions in the abstract, but that the right choice depends heavily on how much data and compute you actually have for pretraining.

For most practitioners working with a modest labelled dataset, for instance the kind of custom photo collection used in part 12's hands-on fine-tuning exercise, the realistic way to use a ViT is to start from weights pretrained on a very large dataset by someone else, exactly the transfer learning idea from part 6, rather than training one from scratch. This is true for CNNs as well, but the gap in data appetite between training from scratch and fine-tuning from pretrained weights tends to be more pronounced for ViTs, precisely because they have fewer built-in assumptions helping them along in the low-data regime.

Making ViTs more data-efficient: DeiT

Shortly after the original ViT paper, a follow-up called DeiT, short for Data-efficient image Transformers, from Touvron and colleagues in 2021, showed that a ViT could be trained to strong accuracy on a single, more modestly sized labelled dataset, without the very large-scale pretraining the original paper relied on. The key ingredient was not a change to the core architecture but a change to the training recipe: careful data augmentation, regularisation, and a technique called distillation through attention, where the ViT is trained partly to match the predictions of an already-trained CNN teacher model, using a second, dedicated distillation token added to the sequence alongside the class token.

The broader lesson from DeiT is useful beyond its specific recipe: the inductive bias gap between CNNs and transformers can be partly compensated for with smarter training rather than only with more raw data. This mirrors a theme from part 5 on data augmentation, where good augmentation effectively teaches the network invariances it did not start out knowing.

Bringing structure back: hierarchical transformers

A plain ViT also has a practical cost that is easy to overlook: self-attention compares every token against every other token, so its computation and memory use grow quadratically with the number of tokens. At 196 patches for a 224 by 224 image this is manageable, but higher-resolution images, or applications like detection and segmentation that benefit from finer-grained feature maps, push the token count much higher and quickly make a plain global self-attention layer expensive.

One influential answer to this problem is the Swin Transformer, introduced by Liu and colleagues in 2021, which restores some CNN-like structure without going back to convolutions. Instead of letting every patch attend to every other patch across the whole image, Swin restricts self-attention to small local windows, similar in spirit to a convolution's local receptive field, which keeps computation manageable. To avoid the obvious downside of windows that never talk to each other, Swin shifts the window boundaries between consecutive layers, so that patches that were in separate windows in one layer end up sharing a window in the next, letting information gradually flow across the whole image over several layers. Swin also merges neighbouring patches together as the network goes deeper, building a multi-scale, pyramid-like feature hierarchy much like a CNN's progressively downsampled feature maps, which makes it a natural drop-in backbone for detection and segmentation heads of the kind covered in parts 7 and 8.

The general pattern worth remembering is that pure global self-attention and pure convolution sit at two ends of a spectrum, and a great deal of productive research since the original ViT paper has explored useful middle grounds: local windows, hierarchical downsampling, and hybrid designs that use a small convolutional stem before switching to transformer blocks. None of this contradicts the core self-attention idea explained earlier in this article, it is the same mechanism applied more carefully to control its cost and to reintroduce helpful structure.

ViT or CNN: how to decide in practice

There is no universally correct answer, but a few practical factors reliably matter when choosing between them for a real project:

• Amount of relevant pretraining data available: if you can start from a ViT checkpoint pretrained on a very large, broadly relevant dataset, the data-hunger concern mostly disappears, since you are fine-tuning, not training from scratch.
• Size of your own labelled dataset: with a small custom dataset and no suitable large ViT checkpoint, a pretrained CNN such as a ResNet, as used in part 12, is usually the safer and more reliable starting point.
• Compute budget at inference time: plain ViTs process a fixed-length sequence of patches regardless of image content, and their memory use grows quickly with image resolution because of the quadratic cost of self-attention, which matters for deployment on constrained hardware.
• What downstream task you need: hierarchical transformers and hybrid backbones tend to be better suited to detection and segmentation than a plain classification-only ViT, because they naturally produce multi-scale feature maps.
• How much you value long-range reasoning across the whole image: tasks where distant parts of an image are mutually informative, for example recognising a scene from scattered contextual clues rather than one localised object, are exactly where self-attention's unrestricted global view has a natural advantage over a CNN's local filters.

In practice, a lot of current work does not pick one extreme or the other, it mixes convolutional stems with transformer blocks, or uses windowed attention that behaves like a controllable dial between the two.

Where Vision Transformers show up beyond classification

Part 7 of this path already introduced DETR, an object detector that uses a transformer decoder to turn a set of learned queries into bounding boxes, attending over features extracted by a CNN backbone. The architecture described in this article, the ViT encoder itself, is increasingly used as a drop-in replacement for that CNN backbone, feeding patch token features into the same kind of detection or segmentation heads discussed in parts 7 and 8, rather than feeding convolutional feature maps.

ViTs are also central to several ways of pretraining vision models without labelled data, a topic the next part of this path, part 10, covers in depth. One widely used idea, masked image modelling, works by hiding a large fraction of an image's patches and training the network to reconstruct them, directly reusing the patch-and-position-embedding structure explained here, since patches are a natural unit to mask and predict. This article has intentionally stopped short of explaining that training method itself, since that is exactly where part 10 picks up.

ViT-style encoders also appear in vision-language models, for example CLIP from Radford and colleagues in 2021, which pairs an image encoder with a text encoder and trains both together so that matching image-text pairs end up with similar representations. The patch-based, sequence-of-tokens design explained in this article is exactly what makes it straightforward to plug a transformer image encoder into a system that otherwise speaks the language of sequences, since images and text can be represented in a compatible form.

Common mistakes and things to watch for

• Training a ViT from scratch on a small dataset and expecting CNN-like results. Without the right scale of pretraining data or a data-efficient recipe like DeiT's, a plain ViT trained from scratch on a small dataset usually underperforms a comparable pretrained CNN, simply because it has to learn spatial structure that a CNN gets for free.
• Changing input resolution without handling position embeddings correctly. Position embeddings are tied to a specific grid size, for example 14 by 14 for 224-pixel images with 16-pixel patches. Feeding in a different resolution changes the number of patches, and the pretrained position embeddings will no longer match unless they are properly resized or interpolated, something most library implementations handle automatically but that silently breaks custom code that is not aware of it.
• Forgetting that patch size must evenly divide the input size. If it does not, implementations either crop, pad, or throw an error, and the resulting mismatch is easy to miss until results look oddly degraded.
• Assuming attention weights are a reliable explanation of what the model is looking at. Visualising attention maps is tempting and often informative, but, much like the saliency methods discussed in part 11, attention weights are not a guaranteed, fully faithful account of which input regions actually caused a given prediction, since information from many tokens gets mixed across many layers.
• Using a plain ViT where a hierarchical backbone would suit the task better, particularly for detection and segmentation, where multi-scale features matter and a fixed-resolution, single-scale set of patch tokens from a plain ViT is a less natural fit without extra adaptation.
• Not budgeting for the quadratic cost of self-attention when increasing image resolution. Doubling the resolution along each dimension quadruples the number of patches, and self-attention's cost grows roughly with the square of the number of tokens, so compute and memory use can grow faster than expected.

Summary and what comes next

Vision Transformers reframe an image as a sequence: cut it into fixed-size patches, turn each patch into a vector with a single linear projection, add a learnable position embedding so the model knows where each patch came from, prepend a class token to summarise the whole image, and let a stack of self-attention layers allow every patch to exchange information with every other patch, regardless of distance. This gives up the locality and translation equivariance that CNNs get for free, which is why plain ViTs tend to need large amounts of pretraining data or carefully designed training recipes such as DeiT's distillation approach to match CNN performance on smaller datasets. Hierarchical designs such as the Swin Transformer bring back some convolution-like structure, through local windows and multi-scale pyramids, to control the cost of self-attention and to serve as better backbones for detection and segmentation.

The worked numeric example in this article showed self-attention doing one concrete thing: turning each token into a weighted average of every token's value vector, with weights learned from similarity between queries and keys. That one mechanism, repeated across many heads and many stacked layers, is the entire engine behind every transformer architecture, whether it is reading a sentence or looking at a photograph.

Part 10 builds on the patch-based structure introduced here to explain how vision models can be pretrained without any labels at all, through contrastive learning and masked image modelling, both of which lean directly on the idea of patches as tokens established in this article. After that, part 11 turns to interpretability, asking what a trained network, CNN or transformer, is actually looking at when it makes a prediction, before the path moves into fully hands-on projects in parts 12 through 14.