Vision Transformers (ViT)
Introduction
For nearly a decade, convolutional neural networks (CNNs) were the undisputed default for computer vision. The Vision Transformer, introduced by Dosovitskiy et al. in 2020 (“An Image is Worth 16×16 Words”), showed that you can throw away convolutions entirely and apply the very same Transformer encoder used for language — with almost no vision-specific modifications — directly to images.
The trick is deceptively simple: a Transformer expects a sequence of tokens, so we just turn an image into a sequence of tokens. We chop the image into a grid of fixed-size, non-overlapping patches, treat each patch as a “word,” and let self-attention do the rest. That is the entire idea behind the title.
One-sentence summary: split an image into patches → linearly embed each patch into a token → add positional embeddings → prepend a learnable [CLS] token → feed the sequence into a standard Transformer encoder → read the class from the [CLS] token's output.
From Image to Tokens, Step by Step
1. Split the image into patches
An image of size H × W is divided into non-overlapping square patches of size p × p. This produces N = (H/p) · (W/p) patches. For the original paper, a 224×224 image with p = 16 gives N = (224/16)·(224/16) = 14·14 = 196 patches. With p = 16 the picture is literally described as 16×16 “words.”
2. Flatten & linearly embed each patch
Each patch contains p · p · C values (C = color channels, e.g. 3 for RGB). Flatten it into a vector and pass it through a single shared linear layer (the patch embedding) to produce a D-dimensional token. In practice this linear projection is implemented as a convolution with kernel size = stride = p, which is just a tidy way of doing “cut into patches and project” in one operation.
3. Add positional embeddings
Self-attention is permutation-invariant — it has no built-in sense of where a patch sits in the image. So we add a learnable positional embedding to each patch token, giving the model the spatial information (e.g. “top-left” vs. “bottom-right”). ViT uses standard 1-D learnable position embeddings; interestingly they still recover 2-D image structure during training.
4. Prepend a learnable [CLS] token
Borrowing the idea from BERT, ViT prepends one extra learnable token, the [CLS] token, to the front of the sequence. It is not derived from any patch. As it flows through the encoder it attends to all patches and aggregates a global summary of the image. The sequence length becomes N + 1.
5. Run the Transformer encoder, read [CLS]
The whole sequence goes through a standard Transformer encoder: alternating multi-head self-attention and MLP blocks, each with residual connections and layer normalization. After the final layer, the output vector at the [CLS] position is fed into a small MLP classification head to produce the class probabilities. The patch tokens' outputs are typically not used for classification — only the [CLS] token is.
The attention each layer computes:
Attention(Q, K, V) = softmax(Q Kᵀ / √dₖ) V
Q, K, V are linear projections of the token embeddings (queries, keys, values); dₖ is the key dimension and the √dₖ scaling keeps the dot-product logits from exploding. Crucially, every token can attend to every other token in a single layer — so a patch in the top-left can directly influence a patch in the bottom-right from layer 1.
Interactive: Image → Patches → Attention
This is the heart of ViT. Choose a patch size and watch the image get carved into a sequence of patch tokens, with the count N = (H/p)·(W/p) updating live. Then pick a query patch and reveal the attention heatmap: brighter red means more attention mass. The two bright disks in the image are far apart yet look alike — select one and turn on attention to see self-attention link them across the whole image, something a small CNN receptive field cannot do.
Click any patch to make it the query. The cyan grid shows the 6 × 6 patch tiling.
The red overlay is the attention distribution from the selected query patch to every patch (softmax-normalized, sums to 1). Notice it can light up patches that are far away — self-attention sees the whole image from layer 1, unlike a small CNN filter.
These tokens (x₁ … xN), plus the prepended learnable [CLS] token, get positional embeddings added and are fed into a standard Transformer encoder.
The attention weights here are a plausible illustration computed from patch appearance similarity and passed through a real softmax (so they sum to 1), mirroring how a trained head behaves; they are not from a trained network. The patchification math, however, is exact.
ViT vs. CNNs: Inductive Bias
The deepest difference between ViTs and CNNs is inductive bias— the assumptions baked into the architecture before any data is seen. CNNs assume a lot about images; ViTs assume very little.
| Property | CNN | Vision Transformer |
|---|---|---|
| Locality | Built in: small filters only see a local neighborhood | None: attention is global from layer 1 |
| Translation equivariance | Built in via weight-shared convolutions | Not enforced; must be learned from data |
| Weight sharing | Same filter slides over all positions | Shared patch-embedding & attention params, but no spatial filter sharing |
| Receptive field | Grows gradually with depth | Whole image immediately (global context) |
| Data efficiency | Strong on small/medium datasets | Data-hungry; needs large-scale pretraining |
| Scaling behavior | Gains taper with more data | Keeps improving; scales better with data & compute |
Why CNNs are data-efficient
Their strong inductive biases (locality, translation equivariance, weight sharing) are correct priors for natural images. The network does not have to learn that nearby pixels are related or that a cat is a cat wherever it appears — that knowledge is hard-wired, so it learns well from comparatively little data.
Why ViTs are data-hungry
With weak inductive bias, a ViT must learn these regularities from scratch. On mid-sized datasets like ImageNet alone it underperforms a comparable ResNet. But pretrain it on a huge dataset (the paper used Google's JFT-300M, ~300M images) and it matches or beats the best CNNs — and keeps getting better as data and model size grow.
The key tradeoff: inductive bias is a shortcut. CNNs trade flexibility for sample-efficiency; ViTs trade sample-efficiency for flexibility and superior scaling. Give a ViT enough data and its freedom from hand-coded assumptions becomes an advantage.
Hybrids, Details & What Attention Reveals
Hybrid ViT (CNN + Transformer)
You do not have to embed raw pixel patches. A hybridarchitecture first runs the image through a few CNN layers (e.g. a ResNet stem) and forms the patch tokens from the resulting feature map instead of from raw pixels. This injects a little convolutional inductive bias back in, which helps at smaller data scales. At very large scale, the pure pixel-patch ViT catches up to and surpasses the hybrid.
The patch-embedding = a strided convolution
Steps 1 and 2 (cut into patches, then linearly project) are usually fused into a single Conv2d with kernel size p and stride p. Because stride equals kernel size, the receptive fields do not overlap — each output “pixel” is exactly one embedded patch token. It is a convenient implementation, not a return to convolutional inductive bias.
Attention maps show what the model looks at
Because every prediction is mediated by attention weights, we can inspect them. Averaging attention from the [CLS] token back to the image patches (e.g. via “attention rollout”) produces a saliency-like map that tends to highlight the actual object being classified. This makes ViTs relatively interpretable: the heatmap in the demo above is a stand-in for exactly this kind of visualization.
Common Pitfalls & Gotchas
Watch out for
- • Training a ViT from scratch on a small dataset — it will lose to a CNN.
- • Forgetting positional embeddings — the model becomes blind to spatial order.
- • Quadratic cost: attention scales as O(N²) in the number of patches, so smaller patches (larger N) are far more expensive.
- • Changing image resolution at test time misaligns the learned positional embeddings (needs interpolation).
Good to remember
- • Smaller patches → more tokens → finer detail but more compute.
- • Large-scale pretraining + fine-tuning is the standard recipe.
- • Variants like DeiT add distillation/augmentation to train well on ImageNet alone.
- • The [CLS] token output, not the patch outputs, drives classification.
Key Takeaways
- ViT applies a standard Transformer encoder to images by turning them into a sequence of fixed-size, non-overlapping patches — “an image is worth 16×16 words.”
- Pipeline: patches → linear patch embedding (tokens) → add positional embeddings → prepend learnable [CLS] token → Transformer encoder → classify from the [CLS] output.
- The number of patches is exactly N = (H/p)·(W/p); the sequence fed to the encoder has length N + 1 because of the [CLS] token.
- Self-attention is global from the first layer: any patch can attend to any other patch, capturing long-range context a small CNN receptive field would miss.
- CNNs have strong inductive biases (locality, translation equivariance, weight sharing) making them data-efficient; ViTs have weak inductive bias, so they need large-scale pretraining (e.g. JFT) to win — but they scale better with data and compute.
- Hybrids feed CNN feature maps into the Transformer; the patch embedding is implemented as a stride-p convolution; and attention maps offer a built-in window into what the model looks at.