Language Modeling Fundamentals
What Is a Language Model?
A language model (LM) is, at its core, a probability distribution over sequences of words (or tokens). Given any sequence, it answers a single question: how likely is this string of text? A good model assigns higher probability to fluent, plausible sentences (“the cat sat on the mat”) than to gibberish (“mat the cat on sat the”).
This one idea is the foundation of nearly all modern NLP. Translation, summarization, chatbots, and code completion are all powered by models that learned to assign probabilities to text — and, as a side effect, learned to generate it. Everything from n-gram counts to GPT is a different way of computing the same quantity:
This lesson builds the foundation that GPT-style models stand on. By the end you will understand the chain rule factorization, n-gram models and their limits, why neural models generalize better, how next-token prediction gives us free training labels, and how we measure a model with perplexity.
The Chain Rule: Breaking a Sentence Apart
Estimating the probability of a whole sentence directly is hopeless — there are far too many possible sentences to ever count. Instead, we use the chain rule of probability to factor the joint probability into a product of conditional probabilities, one per word:
Read it left to right: the probability of the whole sequence is the probability of the first word, times the probability of the second word given the first, times the third given the first two, and so on. Each factor P(wₜ | w₁…wₜ₋₁) is a next-word prediction conditioned on everything that came before.
Key insight: The hard problem “score a whole sentence” reduces to the simpler, repeatable problem “predict the next word given the context.” A language model only ever has to learn one thing — P(wₜ | context) — and the chain rule stitches those predictions into a probability for any sequence.
N-gram Models and the Markov Assumption
The full context w₁…wₜ₋₁ grows without bound, so we cannot estimate P(wₜ | w₁…wₜ₋₁) directly. The classic fix is the Markov assumption: pretend that only the last few words matter. An n-gram model truncates the history to the previous n−1 words.
| Model | Approximation | Context used |
|---|---|---|
| Unigram | P(wₜ) | none (ignores order) |
| Bigram | P(wₜ | wₜ₋₁) | previous 1 word |
| Trigram | P(wₜ | wₜ₋₂, wₜ₋₁) | previous 2 words |
Estimating an n-gram model is just counting. For a bigram model we use the maximum-likelihood estimate:
That is: of all the times wₜ₋₁ appeared, what fraction of the time was it followed by wₜ?
The sparsity problem
Most word combinations never appear in the training data, even if they are perfectly valid. Any n-gram with a zero count gets probability 0 — and because the chain rule multiplies the factors, a single zero makes the entire sentence have probability 0 (and infinite perplexity). The number of possible n-grams grows like Vⁿ (V = vocabulary size), so the data is hopelessly sparse for large n.
Smoothing
Smoothing patches this by shaving probability mass off seen events and redistributing it to unseen ones, so nothing is exactly 0. The simplest is add-k (Laplace) smoothing:
Adding k to every count guarantees every word has a small but non-zero probability. The interactive demo below uses add-k smoothing with k = 0.1. More sophisticated schemes (Kneser–Ney, backoff, interpolation) do this far better, but they all fight the same sparsity battle.
From Counting to Neural Language Models
N-gram models treat every word as an atomic symbol with no relationship to any other word. They have no way to know that cat and dog behave similarly, so seeing “the cat ran” teaches them nothing about “the dog ran.” And longer context is impossible: a 5-gram model already needs to estimate up to V⁵ probabilities.
Neural language models fix both problems by learning distributed representations (embeddings): each word becomes a dense vector, and words used in similar contexts end up with similar vectors. The model then computes P(wₜ | context) as a smooth function of those vectors. This buys two huge advantages:
Generalization
Because cat and dog have nearby embeddings, what the model learns about one transfers to the other. Unseen word combinations can still get sensible probabilities — no zero-count cliff.
Long context
Instead of memorizing exact n-grams, the network compresses context into hidden state. RNNs and especially Transformers can condition on hundreds or thousands of previous tokens — the full chain-rule context, not a tiny window.
The chain rule never changed — neural LMs still factor P(w₁…wₙ) = Πₜ P(wₜ | w₁…wₜ₋₁). What changed is how each conditional is computed: a learned neural function instead of a lookup table of counts.
Next-Token Prediction: Self-Supervised Learning
How do you train a model to predict the next word? You do not need humans to label anything — the labels are already in the text. This is self-supervised learning: for every position in a corpus, the “correct answer” is simply the word that actually comes next.
Free labels from raw text. Slide a window across “the cat sat on the mat”:
- input: “the” → target: “cat”
- input: “the cat” → target: “sat”
- input: “the cat sat” → target: “on”
- input: “the cat sat on” → target: “the”
Every position is a free training example. This is exactly why LMs can train on trillions of tokens of internet text without manual labeling — and why scale works so well.
At each step the model produces a score for every word in the vocabulary, then converts those scores (logits z) into a probability distribution with the softmax:
The softmax turns arbitrary real-valued scores into a valid distribution over the whole vocabulary — every value is positive and they sum to 1. Training minimizes the cross-entropy loss, which for a single position is just the negative log probability the model assigned to the true next word:
Minimizing this pushes probability mass onto the words that actually occur. Note the loss goes down as the probability of the correct word goes up — and that exact same quantity, averaged over a held-out test set, is what perplexity measures.
Interactive: Bigram Generator & Perplexity Meter
This is a real bigram model built in your browser from a tiny fixed corpus (counts + add-k smoothing). The left panel generates text by sampling P(next | current) token by token; the right panel evaluates a sentence by computing its perplexity. Both use the exact same probabilities — that is the whole point: predicting the next word powers both generation and scoring.
1. Generate text token-by-token
The model only knows P(next word | current word). Pick a starting word, then sample forward. The bar chart shows the candidate distribution at the current step.
2. Score a sentence with perplexity
Perplexity = exp(average negative log-likelihood). Fluent, corpus-like sentences get low perplexity; word-salad gets high perplexity.
A perplexity of 2.3 means the model is, on average, as uncertain as if it were choosing uniformly among ~2.3 words at each step.
Evaluation: Perplexity
How do we say one language model is better than another? We check how well it predicts held-out text it never trained on. The standard metric is perplexity, and it is defined directly from the cross-entropy loss above. First, the average negative log-likelihood (NLL) over a sequence of N tokens:
Perplexity is simply the exponential of that average — equivalently, the exponential of the cross-entropy:
Branching factor intuition. Perplexity is the model's effective average number of choices at each step. A perplexity of 10 means the model is, on average, as confused as if it were picking uniformly among 10 equally likely words. A perfect model that always assigns probability 1 to the right word has perplexity 1. Lower is better.
Bits-per-token connection
If you measure cross-entropy with log₂ instead of the natural log, you get bits per token — the average number of bits needed to encode each token under the model. The two are linked by a clean identity:
So a model at 3 bits/token has perplexity 2³ = 8. Lower bits-per-token means a better compressor of the language, which means a better predictor.
Watch out for
- A single zero-probability token sends perplexity to ∞ — always smooth, and use the same tokenization when comparing models.
- Perplexity is only comparable across models that share a vocabulary and tokenizer. Word-level and subword (BPE) perplexities are not directly comparable.
- Lower perplexity correlates with better language modeling, but it is not a perfect proxy for downstream task quality or helpfulness.
N-gram vs. Neural Language Models
| Aspect | N-gram model | Neural LM (incl. Transformers) |
|---|---|---|
| Word representation | Atomic symbols (no similarity) | Dense embeddings (similar words cluster) |
| Context length | Fixed, tiny (n−1 words) | Long (hundreds–thousands of tokens) |
| Unseen combinations | Zero probability (needs smoothing) | Generalizes from similar contexts |
| Training | Count and normalize | Gradient descent on cross-entropy |
| Typical perplexity | High (poor generalization) | Much lower (state of the art) |
The next lesson on GPT builds directly on everything here: it is a neural language model that computes P(wₜ | w₁…wₜ₋₁) with a Transformer decoder, trains by next-token prediction on massive corpora, and is evaluated with — you guessed it — perplexity.
Key Takeaways
- A language model is a probability distribution over token sequences: P(w₁, …, wₙ).
- The chain rule factors that joint probability into a product of next-word predictions: P(w₁…wₙ) = Πₜ P(wₜ | w₁…wₜ₋₁).
- N-gram models apply the Markov assumption (keep only the last n−1 words) and estimate probabilities by counting, but they suffer from sparsity and need smoothing to avoid zeros.
- Neural LMs learn distributed embeddings, so they generalize across similar words and handle long context — the chain rule stays the same, only the conditional computation changes.
- Next-token prediction is self-supervised: raw text supplies free labels, and a softmax over the vocabulary turns logits into next-token probabilities trained with cross-entropy.
- Perplexity = exp(average negative log-likelihood) = exp(cross-entropy). It is the model's average branching factor; lower is better, and it equals 2 raised to the bits-per-token.