CLIP & Cross-Modal Alignment
One space for pictures and words
Most vision models are trained to predict a fixed list of labels. CLIP (Contrastive Language–Image Pre-training, Radford et al., 2021) does something different: it learns a single shared embedding space that holds both images and text. In that space, an image and a sentence that describes it land close together, while unrelated images and sentences land far apart.
The training signal is not human-annotated class labels but roughly 400 million image–caption pairs scraped from the web. CLIP never needs someone to enumerate categories — it just needs to know which caption went with which image. From that, it learns representations general enough to classify, search, and condition image generation without task-specific training.
The core trick: instead of predicting a label, predict which caption goes with which image inside a batch. That "matching" objective is enough to align two completely different modalities into the same geometry.
Two encoders, one space
CLIP has two separate networks that never share weights:
- An image encoder (a ResNet or a Vision Transformer) that turns an image into a vector.
- A text encoder (a Transformer) that turns a caption into a vector.
Each encoder ends with a linear projection into the same d-dimensional space, and both outputs are L2-normalized so they sit on the unit hypersphere. On a unit sphere, the dot product of two vectors equals their cosine similarity, which is why CLIP measures alignment with cosine similarity throughout.
image → image_encoder → Iₑ and text → text_encoder → Tₑ, then normalize: Iₑ ← Iₑ / ‖Iₑ‖, Tₑ ← Tₑ / ‖Tₑ‖. Now Iₑ · Tₑ = cos(θ).
The contrastive objective (InfoNCE)
Take a batch of N image–text pairs. Embed all N images and all N captions, then form the N × N similarity matrix S, where Sᵢⱼ = cos(imageᵢ, captionⱼ). There are exactly N correct pairs — the diagonal Sᵢᵢ — and N² − N incorrect pairs — everything off the diagonal.
The objective: make every diagonal entry the largest in its row and its column. Concretely, the logits are the similarities scaled by a learned temperature τ:
Treat each row as a classification problem over the N captions, with the correct class being the diagonal — that is a softmax + cross-entropy. Do the same for each column (classify each caption among the N images). The CLIP loss is the average of the two directions:
This is the InfoNCE loss, applied symmetrically. Minimizing it pulls matching pairs together (raises the diagonal) and pushes mismatched pairs apart (lowers the off-diagonal). The off-diagonal entries act as in-batch negatives, which is why large batch sizes help: more negatives per step sharpen the signal.
Temperature τ: a learned scalar (CLIP stores it as a log value and clamps it) that controls how peaked the softmax is. Small τ makes the model very confident about the nearest match; it is learned jointly with the encoders rather than hand-tuned.
Interactive: the similarity matrix
Below is a tiny batch of 5 images and 5 captions with hand-placed embeddings. The heatmap shows the real cosine-similarity matrix. The diagonal (outlined in green) is what contrastive training drives up; everything else it drives down. Slide the temperature to see how the same similarities turn into a sharper or softer loss.
Contrastive similarity matrix
Each cell is the cosine similarity between an image embedding (rows) and a caption embedding (columns). Contrastive training pulls the diagonal (the N matching pairs) up and pushes the off-diagonal (the N²−N mismatched pairs) down.
What the loss measures. Logits = similarity / τ. Each row is turned into a softmax over captions; the loss is the negative log-probability assigned to the correct (diagonal) caption. The same is done down each column (caption → image). CLIP averages the two directions — a symmetric cross-entropy over the matrix.
Lower τ sharpens the softmax, so even small similarity gaps produce confident, low-loss predictions when the diagonal is the largest entry in its row and column — exactly the structure contrastive training creates.
Zero-shot classification
Because text and images live in the same space, classification needs no trained classifier head. To classify an image into one of K categories:
- Write each class as a text prompt, e.g. "a photo of a {class}" (this prompt template measurably helps).
- Run each prompt through the text encoder to get K text embeddings.
- Run the image through the image encoder to get one image embedding.
- Compute cosine similarity between the image embedding and all K text embeddings, apply a softmax, and predict the highest-scoring class.
The classifier's "weights" are simply the text embeddings of the label prompts — you can swap in a new set of labels at inference time and instantly classify into categories the model was never explicitly trained on.
Interactive: zero-shot prediction
Pick an image and watch CLIP-style classification happen as nearest-text retrieval. The bars are the softmax over cosine similarities to each label prompt.
Zero-shot classification
No classifier head is trained. Pick an image, embed each candidate label as the prompt "a photo of a {class}", and pick the label whose text embedding is closest to the image embedding.
Interactive: the shared space
The shared embedding space (2D projection)
Both encoders map into one vector space. Each image (●) sits next to its matching caption (▲). A dotted line connects each matching pair — short lines mean the alignment pulled them together.
● = image embedding ▲ = caption embedding. Colors pair an image with its caption.
Where CLIP gets used
| Use | How the shared space helps |
|---|---|
| Zero-shot classification | Classify into arbitrary label sets without training, by nearest text embedding. |
| Image / text retrieval | Search images with a text query (or vice versa) by cosine similarity in the shared space. |
| Text-to-image models | CLIP's text encoder is a common conditioning backbone for diffusion / generative models (the prompt is encoded into CLIP space). |
| Data curation & filtering | Score how well an image and caption match to filter noisy web datasets. |
Strengths and limits
Strengths
- Open-vocabulary: any text can be a class.
- Strong, transferable features from cheap web pairs.
- One model serves classification, retrieval, and conditioning.
Limits
- Struggles with fine-grained counting, spatial relations, and OCR-heavy tasks.
- Inherits biases and noise from uncurated web data.
- Prompt wording affects accuracy ("prompt engineering" needed).
Key Takeaways
- CLIP learns a single shared embedding space for images and text via contrastive learning on ~400M web image–caption pairs.
- Two independent encoders (image + text) project, then L2-normalize, into the same space; alignment is measured with cosine similarity.
- The objective is symmetric InfoNCE over the N × N similarity matrix: maximize the N diagonal (matching) similarities and minimize the N² − N off-diagonal (mismatched) ones, with a learned temperature τ.
- Off-diagonal entries are in-batch negatives, so larger batches give a stronger signal.
- Zero-shot classification = embed label prompts ("a photo of a {class}") and pick the nearest text embedding to the image — classification becomes retrieval, with no trained classifier head.
- The aligned space powers retrieval and serves as the text-conditioning backbone for many text-to-image generators.