Sequence Models for NLP
This lesson sits between the Recurrent Neural Networks lesson and the Transformers lesson. It traces the line of ideas — from counting word pairs, to recurrent memory, to gated memory, to attention — that turns a flat bag of words into a model that truly understands sequence.
Why Order Matters
Language is sequential. “The dog bit the man” and “the man bit the dog” use the exact same words, yet mean opposite things. A model that treats text as an unordered bag of words sees these two sentences as identical. To do NLP well, a model must be sensitive to order and to context — which earlier words shape the meaning of later ones.
The central challenge is that sequences have variable length and that important relationships can span long distances. In “The keys to the cabinet are on the table,” the verb must agree with “keys,” not the nearer noun “cabinet.” A good sequence model has to carry the right information across that gap.
Starting Point: n-gram & Markov Models
The earliest statistical language models estimate the probability of a word from the few words just before it. An n-gram model makes a Markov assumption: the next word depends only on the previous n − 1 words.
trigram (n=3): P(wₜ | w₁ … wₜ₋₁) ≈ P(wₜ | wₜ₋₂, wₜ₋₁)
These probabilities are just counts: P(wₜ | wₜ₋₁) = count(wₜ₋₁, wₜ) / count(wₜ₋₁). Simple, fast, and surprisingly useful — but with two crippling limits:
- Tiny context window. A bigram sees one word back; even a 5-gram forgets everything older than four words. Long-range agreement is impossible.
- Sparsity / no generalization. The vocabulary raised to the power n explodes combinatorially. Most n-grams are never observed, so the model assigns them zero probability and has no notion that “cat” and “kitten” are related.
Try the bigram model below. It picks the next word using only the word you select — a vivid illustration of how short its memory is.
P(next | “the”) estimated by counting which words follow “the” in the corpus:
RNNs: A Hidden State That Carries Context
A Recurrent Neural Network (RNN) removes the fixed-window limit by maintaining a hidden state — a vector that acts as a running summary of everything read so far. At each time step it reads one token and updates that state:
ŷₜ = softmax(Wᵧ·hₜ + c)
Where:
- xₜ — the current input token's embedding vector.
- hₜ₋₁ — the hidden state from the previous step (the memory).
- hₜ — the updated hidden state, a blend of new input and past context.
- Wₓ, Wₕ, Wᵧ — weight matrices, shared across every time step.
- tanh — squashes each cell into the range −1…1.
The crucial detail is weight sharing: the same Wₓ and Wₕ are applied at every position, so an RNN handles sequences of any length with a fixed number of parameters. Information flows forward through hₜ, letting the network remember a subject mentioned many words ago.
Interactive: Unroll the RNN
Step through a sentence one token at a time and watch the 4-cell hidden state update via the actual recurrence hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁) with small fixed weights. Notice how each new word both adds information and reshapes the memory of earlier words.
Press Step to read the first token
The hidden state starts as all zeros (h₀). Each step folds in one new token while keeping a compressed memory of everything seen so far.
What to watch: the colored bars are the hidden-state vector hₜ. Red = positive, blue = negative, white ≈ 0. The same vector is reused at every step — that is the single “memory” the RNN carries forward.
Step through a sentence and notice how the pattern set by an early word still tints the state several steps later, but slowly fades because Wₕ has magnitude below 1. That fading is the seed of the vanishing-gradient problem discussed below.
The Vanishing-Gradient Problem
RNNs are trained with backpropagation through time (BPTT): the network is unrolled across the whole sequence and gradients are propagated backward from the end to the beginning. To learn a dependency between a word at step t and one at step t+k, the gradient must travel back through k applications of the same recurrence.
Each backward hop multiplies by the recurrent weight Wₕ and by the derivative of tanh (which is at most 1, and usually much smaller). Chaining k such factors gives a product roughly proportional to a number raised to the k-th power:
Vanishing gradients
If the relevant factors are < 1, the product shrinks toward zero as k grows. The signal from distant words disappears, so the RNN effectively cannot learn long-range dependencies.
Exploding gradients
If the factors are > 1, the product blows up, causing unstable, diverging updates. This one is usually tamed by gradient clipping; vanishing is the harder problem.
In the demo above, Wₕ has magnitude below 1, so you can literally see the influence of an early word fade out of the state — the same decay that, in the backward pass, starves distant words of learning signal.
LSTM & GRU: Gated Memory
Long Short-Term Memory (LSTM) and the simpler Gated Recurrent Unit (GRU) fix vanishing gradients by adding learnable gates that control information flow. Instead of overwriting the state every step, they can choose to keep it almost unchanged — creating a near-uninterrupted path for gradients to flow across many steps.
The LSTM's key innovation is a separate cell state cₜ updated mostly by addition rather than repeated multiplication:
hₜ = oₜ ⊙ tanh(cₜ)
- Forget gate fₜ — decides how much of the old cell state to keep.
- Input gate iₜ — decides how much new candidate information c̃ₜ to write.
- Output gate oₜ — decides how much of the cell state to expose as hₜ.
Because cₜ is carried forward with a gated additive update (when the forget gate stays near 1), the gradient can pass back across long spans without vanishing. A GRU achieves a similar effect with only two gates (update and reset) and no separate cell state, making it lighter and often comparably accurate.
Takeaway: gating turns the RNN's leaky bucket into a controllable memory. LSTMs/GRUs dominated NLP from roughly 2014 to 2017 and powered the first strong neural machine-translation systems.
Encoder–Decoder (seq2seq): Variable-Length In → Variable-Length Out
Many NLP tasks map one sequence to another of a different length: translation, summarization, question answering. The encoder–decoder (seq2seq) framework handles this with two recurrent networks:
Encoder
Reads the entire input sequence and compresses it into a fixed-length context vector (typically its final hidden state).
Decoder
Starts from that context vector and generates the output sequence one token at a time, feeding each prediction back in until it emits an end-of-sequence token.
This decouples input and output lengths and is the blueprint behind neural machine translation. But it has a notorious bottleneck: squeezing an entire sentence into one fixed-size vector loses detail, especially for long inputs. The encoder's final state simply cannot hold everything.
The fix that changed everything: attention (Bahdanau et al., 2014) let the decoder look back at all encoder states at each output step, weighting them by relevance instead of relying on one bottleneck vector. This idea is the hinge to the next era.
Why Attention & Transformers Replaced Recurrence
The 2017 paper “Attention Is All You Need” took attention to its logical conclusion: throw away recurrence entirely. The Transformer processes a sequence using only self-attention, where every token directly attends to every other token. The core operation is:
where Q, K, V are the query, key, and value projections of the tokens and dₖ is the key dimension (the √dₖ term keeps the dot products from growing too large). Two properties make this decisively better than recurrence:
| Property | RNN / LSTM | Transformer |
|---|---|---|
| Computation order | Sequential — step t must finish before t+1 | Parallel — all positions computed at once |
| Path length between two tokens | Up to O(n) steps apart (gradient must travel the whole way) | O(1) — any pair connects in a single attention hop |
| Long-range dependencies | Hard — signal decays (even gated) | Direct — every token sees every other |
| Order information | Implicit in the recurrence | Added explicitly via positional encodings |
Parallelism means Transformers train far faster on modern GPUs (recurrence is inherently serial). Constant path length means long-range dependencies are no harder to learn than short ones — directly solving the problem that vanishing gradients created. The cost is that self-attention is O(n²) in sequence length, but for typical lengths the parallelism and quality wins dominate.
Because self-attention itself is order-blind, Transformers re-inject position with positional encodings — the explicit order signal that recurrence used to provide for free. The next lesson unpacks self-attention and the full Transformer block.
Key Takeaways
- Word order and long-range context are essential to meaning; bag-of-words models throw both away.
- n-gram / Markov models use only a fixed, tiny window and suffer from data sparsity — they cannot generalize or remember far back.
- RNNs carry a hidden state hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁) forward through the sequence, with weights shared across all time steps.
- Vanishing gradients (and, less often, exploding ones) make plain RNNs unable to learn dependencies across many steps.
- LSTMs and GRUs add gates and an additive memory path, letting gradients survive across long spans.
- Encoder–decoder (seq2seq) maps variable-length input to variable-length output but bottlenecks meaning into one context vector — attention relieved that bottleneck.
- Transformers replace recurrence with self-attention, gaining full parallelism and O(1) path length between any two tokens, which is why they now dominate NLP.