Tokenization
Also: tokenizer, subword tokenization, BPE, byte-pair encoding, WordPiece
The first step of almost every NLP pipeline: text is cut into units from a fixed vocabulary and each unit is replaced by its ID. Modern LLMs use subword schemes such as BPE or WordPiece, so common words stay whole while rare words split into reusable pieces.
Example
A whitespace tokenizer turns "Isn't NLP amazing?" into ["Isn't", "NLP", "amazing?"], while a punctuation-aware one gives ["Isn", "'", "t", "NLP", "amazing", "?"]. GPT models use BPE with ~50k tokens; BERT uses WordPiece with ~30k.