Probability & Statistics
Why probability runs through all of ML
Machine learning is, at its heart, reasoning under uncertainty. Data is noisy, samples are finite, and the world is only partly observable. The language we use to describe that uncertainty — and to make principled decisions in spite of it — is probability and statistics. Almost every model you will meet is, underneath the notation, an answer to a probabilistic question: “given what I have seen, what is most likely true?”
A few concrete connections we will build toward in this lesson:
- Loss functions are negative log-likelihoods. Minimizing mean-squared error is the same as assuming Gaussian noise and maximizing likelihood. Cross-entropy is the negative log-likelihood of a categorical model.
- Naive Bayes is a classifier built directly out of Bayes’ theorem.
- The bias–variance tradeoff — the single most important idea for understanding overfitting — is a statement about estimators and their sampling distributions.
Random variables
A random variable X is a quantity whose value is determined by a random outcome — the roll of a die, the height of a randomly chosen person, the label of a randomly drawn image. It is not a single number but a recipe that assigns probabilities to the values it can take.
We describe a random variable by its distribution:
- For a discrete X (finitely or countably many values) we use a probability mass function (pmf) P(X = x), with all probabilities summing to 1: Σₓ P(X = x) = 1.
- For a continuous X we use a probability density function (pdf) f(x). A density is not a probability — P(X = x) = 0 for any single point. Probability is area under the curve: P(a ≤ X ≤ b) = ∫ₐᵇ f(x) dx, and the total area is 1.
ML connection: a classifier’s softmax output is a pmf over classes; a generative model of images or audio defines a pdf over a high-dimensional continuous space. “The model” is a distribution.
Common distributions
| Distribution | Type | pmf / pdf | Mean | Variance |
|---|---|---|---|---|
| Bernoulli(p) | Discrete | P(1)=p, P(0)=1−p | p | p(1−p) |
| Binomial(n, p) | Discrete | C(n,k) pᵏ(1−p)ⁿ⁻ᵏ | np | np(1−p) |
| Uniform(a, b) | Continuous | 1/(b−a) on [a,b] | (a+b)/2 | (b−a)²/12 |
| Normal(μ, σ²) | Continuous | see below | μ | σ² |
The Gaussian / Normal pdf:
f(x) = (1 / (σ√(2π))) · e^(−(x−μ)² / (2σ²))
Here μ is the mean (the center of the bell) and σ is the standard deviation (its width). The 1/(σ√(2π)) factor is exactly what is needed to make the total area equal 1. The Normal shows up everywhere because of the Central Limit Theorem (below), and because it is the maximum-entropy distribution for a fixed mean and variance.
Bernoulli models a single yes/no trial (one coin flip, one click/no-click). Binomial counts the successes in n independent Bernoulli trials. These are the building blocks behind logistic regression and binary classification.
Expectation, variance, and standard deviation
The expectation (or expected value, or mean) E[X] is the long-run average value of X — the probability-weighted center of mass of the distribution:
Continuous: E[X] = ∫ x · f(x) dx
The variance Var(X) measures spread — the expected squared distance from the mean:
The second form, Var(X) = E[X²] − E[X]², is the one you will use constantly — it follows from expanding the square and using linearity of expectation. The standard deviation σ = √Var(X) puts the spread back into the same units as X.
Useful properties: E[aX + b] = a·E[X] + b (linearity), and Var(aX + b) = a²·Var(X) (shifting does not change spread; scaling multiplies it by a²). For independent X and Y, Var(X + Y) = Var(X) + Var(Y) — this is exactly why averaging n independent samples shrinks variance by 1/n.
Conditional probability and Bayes’ theorem
Conditional probability P(A | B) is the probability of A given that B happened. It rescales the world to the cases where B is true:
Rearranging this two ways and equating gives Bayes’ theorem, the engine of learning from evidence. Writing H for a hypothesis and D for observed data:
P(H | D) = P(D | H) · P(H) / P(D)
- P(H) — prior: belief before seeing data.
- P(D | H) — likelihood: how well H explains the data.
- P(H | D) — posterior: updated belief after data.
- P(D) — evidence: a normalizer, P(D) = Σₕ P(D | h)P(h).
Worked example: the base-rate trap
A disease affects 1% of people. A test is 99% sensitive (P(test+ | disease) = 0.99) and 95% specific (P(test− | healthy) = 0.95, so a 5% false-positive rate). You test positive. What is the probability you actually have the disease?
= 0.99·0.01 + 0.05·0.99 = 0.0099 + 0.0495 = 0.0594
P(H | D+) = 0.99·0.01 / 0.0594 ≈ 0.167
Only about 17% — despite a “99% accurate” test! Because the disease is rare, the few true positives are outnumbered by false positives drawn from the huge healthy majority. This base-rate effect is the single most common probability mistake, and it is why calibration and base rates matter so much in real ML deployments. Explore it live:
False-positive rate P(test+ | healthy) = 1 − specificity = 5.0%.
P(D) = 0.99·0.010 + 0.050·0.990 = 0.0594
Naive Bayes classifiers apply this theorem directly: they estimate P(class) and P(featureᵢ | class) from training data, then use Bayes’ rule to compute P(class | features). The “naive” part is assuming features are conditionally independent given the class — wrong in general, but cheap and surprisingly effective for text.
The Central Limit Theorem (interactive)
The Central Limit Theorem (CLT) is one of the most remarkable facts in all of mathematics. It says: take any distribution with a finite mean μ and variance σ² — it can be skewed, bimodal, discrete, whatever — draw n independent samples, and average them. As n grows, the distribution of that sample mean x̄ approaches a Normal distribution, no matter what the original looked like:
Two predictions to watch for in the demo: (1) the histogram of sample means becomes a bell curve centered at μ, and (2) its spread is the population σ divided by √n — averaging more values makes the estimate of the mean more precise, but only as fast as 1/√n (to halve the error you need 4× the data).
Pick a deliberately non-normal source, choose a sample size n, and click “Draw” repeatedly. Watch the blue histogram of sample means tighten around the green μ line and snap onto the red Normal curve that theory predicts.
Blue bars: histogram (as a density) of the sample means you have drawn. Red curve: the Normal distribution the CLT predicts. Green dashed line: the true population mean μ.
Source distribution
A flat, perfectly non-bell-shaped distribution. Every value is equally likely.
Each sample mean averages n draws. Bigger n ⇒ tighter, more bell-shaped histogram.
| Empirical | Theory | |
|---|---|---|
| mean of x̄ | — | 0.5000 |
| std of x̄ | — | 0.1291 |
Theory std = σ / √n = 0.289 / √5 = 0.1291. As you collect more means, the empirical std should converge to this value.
Why ML cares: the CLT is why Gaussian noise assumptions are so often reasonable — measurement error is frequently the sum of many small independent effects. It also underpins error bars, confidence intervals, and why mini-batch gradient estimates (averages of per-example gradients) behave roughly normally around the true gradient.
Sampling, estimators, and their bias and variance
We almost never know a distribution’s true parameters; we only have a finite sample. An estimator is a rule for guessing a parameter from data — for example, using the sample mean x̄ to estimate the true mean μ. Because the sample is random, the estimator is itself a random variable with its own distribution (its sampling distribution — exactly what the CLT demo above is drawing).
We judge an estimator θ̂ of a true value θ by two quantities:
- Bias = E[θ̂] − θ. How far off it is on average. An estimator with zero bias is unbiased.
- Variance = Var(θ̂). How much it jumps around from sample to sample.
Why the (n−1) in sample variance? The estimator s² = (1/(n−1)) Σ(xᵢ − x̄)² uses n−1, not n, because dividing by n systematically underestimates the variance (the data hugs its own sample mean too closely). Dividing by n−1 — Bessel’s correction — makes the estimator unbiased. This is a concrete example of bias in an estimator.
The expected squared error of an estimator decomposes beautifully:
This is the bias–variance decomposition, and it generalizes directly to predictive models. A model that is too simple (e.g. a straight line for curved data) has high bias — underfitting. A model that is too flexible memorizes noise and swings wildly with the training set — high variance, overfitting. The art of ML is trading one against the other to minimize total error.
| High bias (underfit) | High variance (overfit) | |
|---|---|---|
| Model | Too simple / too constrained | Too flexible / too many parameters |
| Symptom | Poor on train AND test | Great on train, poor on test |
| Fix | More features, bigger model | More data, regularization, simpler model |
Maximum likelihood estimation
Maximum likelihood estimation (MLE) is the most common way to fit a model’s parameters. The idea is simple: given observed data, choose the parameters θ that make the data most probable. Assuming the data points are independent, the likelihood is a product, and we usually work with its logarithm (the log-likelihood) to turn products into sums:
Maximizing the log-likelihood is identical to minimizing its negative — and that negative log-likelihood is your loss function. This is the bridge between probability and the loss you actually optimize:
- Assume Gaussian noise around the prediction → MLE gives you mean-squared error. (The exponent of the Gaussian, −(y − ŷ)²/(2σ²), is exactly the squared-error term.)
- Assume a Bernoulli / categorical label → MLE gives you cross-entropy (log) loss.
So why do we assume Gaussian noise? Partly the CLT (errors are sums of many small effects), partly because it is the maximum-entropy choice given only a mean and variance, and partly because it hands us the clean, convex squared-error loss. When you train a regressor with MSE, you are — whether you say so or not — doing maximum likelihood under a Gaussian-noise model.
Key Takeaways
- A random variable is described by a distribution: a pmf (discrete) or a pdf (continuous, where probability is area under the curve).
- Know the staples: Bernoulli p(1−p), Binomial np(1−p), Uniform with variance (b−a)²/12, and the Gaussian pdf (1/(σ√(2π)))·e^(−(x−μ)²/(2σ²)).
- Expectation E[X] is the probability-weighted mean; variance Var(X) = E[X²] − E[X]² measures spread; σ = √Var(X) restores the original units.
- Bayes’ theorem P(H|D) = P(D|H)P(H)/P(D) updates beliefs with evidence; with low base rates even very accurate tests yield modest posteriors.
- The Central Limit Theorem makes the sample mean approach Normal(μ, σ²/n) for almost any source — its spread shrinks only as σ/√n.
- Estimators have bias and variance; expected squared error = bias² + variance, the same tradeoff that governs underfitting vs overfitting.
- Maximum likelihood picks parameters that make the data most probable; its negative log-likelihood is your loss — Gaussian noise ⇒ MSE, categorical labels ⇒ cross-entropy.