Contrastive learning
Also: InfoNCE, contrastive loss
A self-supervised approach in which each example has a positive partner (an augmented view or a paired caption) and the other items in the batch act as negatives. A softmax-based loss such as InfoNCE rewards the positive pair for being the most similar; CLIP and SimCLR are trained this way.
L = −log( exp(sim(a, p)/τ) / Σⱼ exp(sim(a, xⱼ)/τ) )
Example
CLIP embeds a batch of N = 32 images and their 32 captions: the 32 matching pairs on the diagonal of the 32 × 32 similarity matrix are positives, and the other 992 pairs are negatives.