KL divergence
Also: Kullback–Leibler divergence, Kullback-Leibler divergence, relative entropy, KL
D_KL(p ‖ q) = Σ p(x) log(p(x)/q(x)) = H(p, q) − H(p). It is always ≥ 0 and asymmetric, so it is not a true distance; it appears in VAE losses and in minimizing cross-entropy.
D_KL(p ‖ q) = Σₓ p(x) log₂(p(x) / q(x)) = H(p, q) − H(p)
Example
For a fair coin p = (0.5, 0.5) modeled as q = (0.9, 0.1): H(p) = 1 bit, H(p, q) ≈ 1.74 bits, so D_KL(p ‖ q) ≈ 0.74 bits.