Machine Translation
Introduction
Machine translation (MT) is the task of automatically converting text from one natural language (the source) into another (the target) while preserving meaning. It is one of the oldest goals of natural language processing, and its history is a near-perfect tour of how the whole field evolved: from hand-written rules, to statistics over huge bilingual corpora, to neural networks, to the Transformer architecture that now powers systems like Google Translate and DeepL.
The central difficulty is that languages do not line up word-for-word. Words reorder (English "red car" becomes French "voiture rouge"), one word maps to many (or none), idioms must be paraphrased, and meaning depends on context far from the word being translated. A good MT system has to model all of this. The story below is largely the story of how we got machines to handle reordering and long-range context well.
The Evolution of Machine Translation
1. Rule-based MT (1950s–1980s)
Linguists hand-wrote bilingual dictionaries plus grammar and transfer rules: parse the source, transform its syntax tree into the target's structure, then generate words. Accurate for narrow domains, but brittle and enormously labor-intensive: every language pair, every exception, and every idiom had to be encoded by hand.
2. Statistical MT (1990s–2014)
Instead of rules, learn from data. Given millions of human-translated sentence pairs, estimate probabilities. The classic framing is the noisy-channel model: to translate a source sentence f into a target e, pick the e that maximizes P(e | f) ∝ P(f | e) · P(e) (Bayes' rule). Here P(f | e) is the translation model (how well the words correspond) and P(e) is a language model (how fluent the output is). The translation model was learned through word and phrase alignment — figuring out which source words produced which target words — formalized by the IBM Models (1–5), trained with EM. Later phrase-based systems translated chunks of words at a time, which captured local reordering better.
3. Neural MT — seq2seq with attention (2014–2017)
A single neural network learns the whole mapping end-to-end. An encoder RNN/LSTM reads the source into vector(s); a decoder RNN generates the target one token at a time. The breakthrough was attention (Bahdanau et al., 2015): instead of forcing the encoder to cram the entire sentence into one fixed vector, the decoder learns to look back at all source positions at each step. Translation quality jumped, especially on long sentences.
4. Transformer-based MT (2017–present)
The Transformer ("Attention Is All You Need", Vaswani et al., 2017) threw out recurrence entirely and built the network from attention alone. The encoder uses self-attention so every source word can directly see every other; the decoder uses masked self-attention plus cross-attention into the encoder. Because there is no sequential recurrence, training parallelizes across positions, so models train faster and scale far larger. This is the architecture behind essentially all modern MT and large language models.
Encoder–Decoder and the Bottleneck Problem
Almost every modern MT system is an encoder–decoder (also called sequence-to-sequence, or seq2seq). The encoder turns the source sentence into a representation; the decoder produces the target sentence from that representation, generating one token at a time and feeding each generated token back in (autoregressive decoding).
In vanilla seq2seq (no attention), the encoder compresses the entire source sentence into a single fixed-length vector — the final hidden state — and the decoder must reconstruct the whole translation from just that one vector. This is the fixed-bottleneck problem: a 3-word sentence and a 40-word sentence both have to fit through the same narrow pipe. Information from early words gets overwritten, and long sentences degrade badly.
How attention fixes it
Attention removes the bottleneck: the encoder keeps a vector for every source position, and at each decoding step the decoder computes a weighted average of those source vectors — a context vector tailored to the word it is about to produce. The weights are a probability distribution over source positions (they sum to 1), so the decoder can effectively "point at" the source words most relevant to the current output word.
The encoder no longer has to cram everything into one vector; the decoder reaches back into the full source whenever it needs to. As a bonus, those attention weights form a soft alignment — a learned, probabilistic version of the word alignments that statistical MT estimated explicitly.
In the Transformer this mechanism is cross-attention. The decoder forms a query from its current state; the encoder outputs supply the keys and values. The attention weights and context are computed as:
where Q are queries (from the decoder), K and V are keys and values (from the encoder for cross-attention), dₖ is the key dimension, and the √dₖ scaling keeps the dot products from growing too large. The softmax(Q Kᵀ / √dₖ) term is the attention-weight matrix visualized below; multiplying by V produces the context vector.
Interactive: Attention Alignment Visualizer
This is the heart of neural translation. Pick a sentence pair and hover a target word to see which source words it attends to, shown both as a heatmap (rows sum to 1) and as connecting lines whose opacity equals the attention weight. Notice how the active source word changes for every output word, and how the lines cross when languages reorder words (e.g. English adjective-before-noun vs. French noun-before-adjective). Turn on step-through to watch the decoder generate one word at a time.
Hover a target word (bottom row) to see which source words it attends to. Brighter cells / more opaque lines = higher attention weight.
| the | red | car | |
|---|---|---|---|
| la | 0.80 | 0.05 | 0.15 |
| voiture | 0.08 | 0.04 | 0.88 |
| rouge | 0.06 | 0.86 | 0.08 |
French puts the adjective AFTER the noun: "red car" becomes "voiture rouge". Watch how the target words attend to source positions in a crossed, reordered pattern.
The alignment matrices here are hand-crafted but plausible illustrations. In a real model they are learned and computed on the fly via softmax(Q Kᵀ / √dₖ), and there are many attention heads, but the qualitative picture — a different soft alignment per output word — is exactly what trained models produce.
Subword Tokenization (BPE)
Real text contains rare words, names, typos, and morphologically rich forms that a fixed word vocabulary cannot cover — they become "unknown" tokens, which MT cannot translate. The fix is subword tokenization, most commonly Byte-Pair Encoding (BPE).
BPE starts from individual characters and greedily merges the most frequent adjacent pair over and over, building a vocabulary of subword units. Common words end up as single tokens, while rare words break into reusable pieces. For example unhappiness might become un + happi + ness, and a novel word like translatable can still be represented from known fragments. This gives an open vocabulary with a fixed-size token set: no word is ever truly out-of-vocabulary, because in the worst case it can be spelled out from characters.
Subword units are why modern MT and LLMs handle rare names, code, and inflected languages gracefully. Variants you may see: WordPiece (BERT) and the byte-level/unigram tokenizers used by GPT and SentencePiece.
Evaluating Translation: BLEU
How do you score a translation when many different wordings are correct? BLEU (Bilingual Evaluation Understudy) is the classic automatic metric. It compares the machine output (the candidate) against one or more human reference translations using n-gram precision: what fraction of the candidate's n-grams (for n = 1, 2, 3, 4) also appear in a reference?
Two important details make BLEU correct:
- Modified (clipped) precision. A candidate can't game the score by repeating a word: each n-gram's count is clipped to the maximum number of times it appears in any single reference. So "the the the the" scores poorly even though "the" is in the reference.
- Brevity penalty (BP). Precision alone rewards short output (an output of one correct word would be 100% precise). BLEU multiplies by a brevity penalty that punishes candidates shorter than the reference: BP = 1 if the candidate length c ≥ reference length r, and BP = e^(1 − r/c) if c < r.
The final score combines them as the brevity penalty times the geometric mean of the clipped n-gram precisions:
where pₙ is the modified precision for n-grams of size n and wₙ are weights (usually 1/4 each for n = 1..4). BLEU ranges from 0 to 1 (often reported ×100). It is fast and correlates reasonably with human judgment in aggregate, but it is imperfect: it ignores meaning and synonyms, so newer metrics like chrF, METEOR, and learned metrics like COMET and BERTScore are increasingly used alongside it.
Common misconception: BLEU measures n-gram precision (how much of the candidate is in the reference), not recall. The brevity penalty is the only thing stopping the model from being rewarded for cherry-picking a few high-precision words and stopping early.
Era Comparison
| Approach | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Rule-based | Hand-written dictionaries + grammar/transfer rules | Predictable; strong in narrow domains | Brittle; huge manual effort per language pair |
| Statistical (SMT) | Word/phrase alignment + noisy channel, learned from corpora | Data-driven; scaled to many pairs | Disfluent; weak long-range reordering; many components |
| Neural seq2seq + attention | Encoder–decoder RNN with attention, end-to-end | Fluent; handles reordering via soft alignment | Sequential RNN is slow; limited long-range memory |
| Transformer | Self-attention encoder + cross-attention decoder | Parallel training; best quality; scales hugely | Compute/data hungry; quadratic attention cost |
Key Takeaways
- MT evolved from hand-written rules, to statistics over bilingual corpora, to neural networks, to the Transformer.
- Statistical MT used the noisy-channel model P(e | f) ∝ P(f | e) · P(e) with explicit word/phrase alignment (IBM models).
- Neural MT is an encoder–decoder: the encoder reads the source, the decoder generates the target autoregressively.
- Attention solved vanilla seq2seq's fixed-bottleneck problem — the decoder attends to all source positions instead of relying on one compressed vector.
- In the Transformer, the decoder attends to the encoder via cross-attention, softmax(Q Kᵀ / √dₖ) V, producing a different soft alignment per output word.
- Subword tokenization (BPE) gives an open vocabulary so rare words are split into reusable pieces rather than dropped.
- BLEU scores translation with brevity-penalized, clipped n-gram precision — a precision metric, imperfect but fast and widely used.