Perplexity
Also: PPL
The standard intrinsic metric for language models: exp of the average negative log-likelihood per token. It reads as the effective number of equally likely choices the model faces at each step, and is only comparable between models that share a tokenizer.
PPL = exp( −(1/N) Σₜ log P(wₜ | w₁…wₜ₋₁) )
Example
A model averaging 3 bits per token has perplexity 2³ = 8: as uncertain as picking uniformly among 8 words. A perfect model has perplexity 1.