Cross-entropy

Also: cross-entropy loss, log loss, categorical cross-entropy

H(p, q) = −Σ p(x) log q(x): the cost of describing data from p using model q. With a one-hot label it reduces to −log of the probability the model gave the correct class.

H(p, q) = −Σₓ p(x) log q(x)

Cross-entropy loss −log p: confident wrong answers cost the most

Example

If the model gives the true class probability 0.7, the loss is −ln 0.7 ≈ 0.36; at 0.99 it is ≈ 0.01, and at 0.01 it is ≈ 4.6.

See it in the lesson: Information Theory →

Related terms

More in Probability & Information