GPT & Decoder Models

The Generative Half of the Transformer

GPT — Generative Pre-trained Transformer — is a decoder-only model. It reads text left to right and learns to do one deceptively simple thing: predict the next token. Scale that objective up over enormous amounts of text and you get models that can write essays, answer questions, translate, and even learn new tasks from a few examples in their prompt.

Where BERT (an encoder) reads the whole sentence at once to understand it, GPT reads only what came before each position so it can generate what comes next. The single architectural trick that makes this possible is the causal attention mask.

Key idea: a GPT block is the Transformer's decoder with the encoder removed. It has masked self-attention and a feed-forward network — but no cross-attention, because there is no encoder to attend to. GPT is decoder-only.

Causal (Masked) Self-Attention

Self-attention computes, for every position, a weighted blend of the other positions' values. The standard formula is:

Attention(Q, K, V) = softmax( (QKᵀ / √dₖ) + M ) V

Here Q, K, V are the query, key and value projections of the input; dₖ is the key dimension (the √dₖ keeps the dot products from getting too large). The new piece is M, the causal mask: an upper-triangular matrix that is 0 on and below the diagonal and −∞ above it.

Because softmax(−∞) = 0, adding −∞ to every "future" score (j > i) zeroes out those weights after the softmax. So position i ends up attending only to positions 0 … i — itself and the past — and its attention weights still sum to 1 over that visible window.

Why mask before softmax? If you zeroed the future weights after softmax instead, the remaining weights would no longer sum to 1, and the model would still have used future information to set the denominator. Masking the logits with −∞ keeps the distribution honest and normalized over the past only.

See It: Stepping Through Generation

The matrix below is the attention pattern for a single masked self-attention layer. Click Generate next token to reveal positions one at a time, just as an autoregressive model does at inference. Notice how each new row can attend to everything before it but nothing after — and toggle the mask off to watch it become bidirectional (BERT-style), which would let generation "cheat" by reading the answer.

Interactive: Causal Attention Mask

Rows are queries (the position attending); columns are keys (positions attended to). Step through generation to reveal each new token's row. With the causal mask on, future cells (to the right of the diagonal) are blacked out — a position can only attend to itself and the past. The visible cells in each revealed row are real softmax weights and sum to 1.

Causal mask ON (GPT)Positions revealed: 1 / 6

Hover any cell to inspect its query→key attention weight.

Causal (GPT): each row only sees the diagonal and left of it. Position i never peeks at i+1…n, so the model can be trained to predict the next token without "cheating."

Training: Next-Token Prediction

GPT is trained with language modeling: given a sequence, predict each token from the ones before it. Thanks to the causal mask, one forward pass over a length-T sequence produces T next-token predictions in parallel — position i predicts token i+1 while only seeing tokens 0 … i. The loss is the average cross-entropy of those predictions:

ℒ = − (1/T) Σ ᵢ log P( xᵢ₊₁ | x₁ … xᵢ )

Each P(xᵢ₊₁ | x₁ … xᵢ) comes from a softmax over the whole vocabulary at position i. Minimizing this cross-entropy pushes the model to put probability mass on the token that actually came next in the training text. No labels are needed — the text is its own supervision (self-supervised learning), which is why GPT can train on web-scale corpora.

Why the mask is non-negotiable here: without it, position i could simply look at token i+1 and copy it — driving the loss to zero while learning nothing. Causal masking forces the model to actually predict.

GPT (Decoder) vs. BERT (Encoder)

Both are Transformer stacks, but they differ in which direction information flows and what they are trained to do.

Aspect	GPT (decoder-only)	BERT (encoder-only)
Attention	Causal / masked (looks left only)	Bidirectional (looks both ways)
Pre-training objective	Next-token prediction (autoregressive LM)	Masked-LM: predict randomly masked tokens
Direction	Unidirectional (left → right)	Non-directional (full context)
Best at	Generation, completion, chat, few-shot	Classification, NER, retrieval, understanding
Can generate fluently?	Yes — that is its whole design	Not naturally (no autoregressive objective)

Watch the terminology. The original Transformer's decoder also had a cross-attention layer that looked at an encoder's output (for translation). GPT keeps the masked self-attention but drops cross-attention entirely. That is what "decoder-only" means.

Stacking Decoder Blocks

A GPT is just the same decoder block repeated many times (GPT-2 small: 12; GPT-3: 96). Each block transforms the sequence of token representations into a richer one, using two sublayers wrapped in residual connections and layer normalization:

Masked multi-head self-attention — mixes information across positions, but only from the past.
Position-wise feed-forward network (FFN) — two linear layers with a non-linearity (typically GELU) applied to each position independently.

Each sublayer is wrapped as x → x + Sublayer(LayerNorm(x)) (modern GPTs use this "pre-norm" arrangement). The residual connection lets gradients flow through deep stacks, and LayerNorm keeps activations well-scaled.

tokens → embeddings + positional encoding
  ↓ (repeat N decoder blocks)
  a = x + MaskedSelfAttn(LayerNorm(x))
  x = a + FFN(LayerNorm(a))
  ↓
final LayerNorm → linear → softmax over vocabulary

How Generation Works at Inference

At inference GPT runs an autoregressive loop: feed in the prompt, read the next-token distribution at the last position, pick a token (greedy, top-k, top-p / nucleus, or temperature sampling), append it, and repeat. Each generated token becomes part of the input for the next step.

Run the prompt through the stack.
Take the logits at the final position → softmax → probability over the vocabulary.
Sample or argmax a token; append it to the sequence.
Repeat until an end-of-text token or a length limit.

KV cache: naively, each new token would re-process the entire sequence — O(T²) work that repeats. Because the causal mask means past positions never depend on future ones, their keys and values never change. GPT therefore caches the K and V vectors of all previous tokens and only computes attention for the new token, turning generation into roughly linear-per-step work. The same mask that enables training also makes fast incremental decoding possible.

Emergence: In-Context & Few-Shot Learning

Something surprising happens as next-token predictors grow: they start to learn from their prompt without any weight updates. Show a large GPT a few worked examples ("sea → mer, dog → chien, cat → ___") and it completes the pattern. This is in-context learning, and the few-example version is few-shot prompting.

Nothing in the training objective explicitly asks for this — it emerges from optimizing next-token prediction at scale. To predict the next token well across the whole internet, the model has to internalize patterns, analogies, and task structure, which at inference it can reuse on tasks it was never explicitly trained for. These emergent abilities tend to appear rather abruptly as model and data scale increase.

The throughline: one objective (predict the next token), one architectural constraint (the causal mask), and a lot of scale give you fluent generation, fast cached decoding, and emergent few-shot learning — all from the decoder half of the Transformer.

Key Takeaways

GPT is decoder-only: masked self-attention + FFN with residuals and LayerNorm, and no encoder cross-attention.
Causal masking adds −∞ to future scores before softmax, so each position attends only to itself and earlier positions; the surviving weights still sum to 1.
The training objective is next-token prediction (cross-entropy language modeling), which is self-supervised and scales to massive text. The mask prevents the model from cheating by reading ahead.
Versus BERT: GPT is unidirectional and generative; BERT is bidirectional and trained with masked-LM for understanding, and does not generate naturally.
Generation is an autoregressive loop; the KV cache exploits the same causality to make each step cheap.
At scale, in-context / few-shot learning and other emergent abilities appear without any change to the objective.