Naive Bayes
Introduction
Naive Bayes is a family of simple, fast, probabilistic classifiers built directly on Bayes' theorem. Given some features of an input (for example, the words in an email), it asks a very natural question: of all the classes I know about, which one most likely produced this input? Despite a wildly unrealistic simplifying assumption, it remains a go-to baseline for text classification and spam filtering, often competitive with far more complex models.
It is a generative classifier: instead of drawing a decision boundary directly, it models how each class generates features, then uses Bayes' rule to invert that into a prediction. Training is essentially just counting, so it is extremely cheap.
Bayes' Theorem
For a class y and a feature vector x = (x₁, x₂, …, xₙ), Bayes' theorem relates the quantity we want — the posterior P(y|x) — to things we can estimate from data:
Because the evidence P(x) is the same for every class, it does not affect which class wins the argmax. We can drop it and work with a proportionality:
- P(y) — the prior: how common class y is before seeing any features (e.g. the fraction of emails that are spam).
- P(x | y) — the likelihood: how probable these exact features are if the class really is y.
- P(y | x) — the posterior: our updated belief in class y after seeing the features. This is what we predict from.
- P(x) — the evidence: a normalizing constant, identical across classes, so we can ignore it for ranking.
The “Naive” Assumption
The hard part is the likelihood P(x₁, …, xₙ | y). For text, x is a high-dimensional vector of word features, and estimating the joint probability of every possible combination of words is hopeless — we would never have enough data. The naive assumption rescues us: it assumes that, given the class, the features are conditionally independent of one another. This factorizes the joint likelihood into a simple product:
Substituting back into Bayes' rule gives the full Naive Bayes model:
This independence claim is almost always false — in real language “free” and “prize” co-occur far more than independence would predict. That is exactly why it is called naive. The payoff is that we only need to estimate each P(xᵢ | y) on its own, which we can do reliably from modest data by counting.
Prediction: the MAP Rule
To classify, we pick the class with the highest posterior — the maximum a posteriori (MAP) estimate:
Multiplying many probabilities (each ≤ 1) drives the product toward zero fast. With hundreds of words, the result can be smaller than the smallest number a computer can represent — it underflows to 0, and all class scores become an indistinguishable 0. The standard fix is to work in log space. Because log is monotonically increasing, the argmax is unchanged, and products become sums:
Logs turn a vanishingly small product into a comfortable sum of moderate negative numbers — numerically stable, and faster (addition beats multiplication). To recover an actual probability P(y|x) for display, exponentiate and normalize across classes (a softmax), using the log-sum-exp trick for stability. The interactive demo below does exactly this.
Laplace (Additive) Smoothing
Suppose a word never appeared in any spam message during training. Then its raw count is 0, so P(word | spam) = 0. In the product form that single zero annihilates the entire spam score — one unseen word vetoes the class, no matter how much other evidence points to spam. This is the zero-frequency problem.
Laplace (additive) smoothing fixes it by pretending we saw every vocabulary word α extra times. For Multinomial Naive Bayes the smoothed likelihood is:
- count(w, c) — times word w occurs across all class-c documents.
- N_c — total token count in class c (the sum of all word counts for that class).
- |V| — vocabulary size; adding α to every word adds α·|V| to the denominator so the probabilities still sum to 1.
- α — the smoothing strength; α = 1 is classic “add-one” smoothing.
No probability is ever exactly 0, so a single unfamiliar word can no longer veto a class. In the demo, slide α toward 0 and watch the math: unseen words push log P(w|c) toward −∞ and destabilize the score.
Interactive Spam Classifier
This classifier is trained live in your browser on a tiny corpus of toy ham and spam messages using Multinomial Naive Bayes with Laplace smoothing. Type or edit a message and watch the per-word evidence, the running log-odds, and the posterior P(spam) vs P(ham) update in real time. Try sliding the smoothing α toward 0 to see the zero-frequency problem bite.
α = 0 means no smoothing (an unseen word makes its class probability collapse). α = 1 is classic add-one smoothing.
Posterior P(class | message)
Per-word contribution to log-odds
Each bar is log P(wᵢ|spam) − log P(wᵢ|ham). Bars to the right (red) push toward spam; bars to the left (green) push toward ham. The running sum (plus the prior) decides the verdict.
* word not seen in training; smoothing gives it a small non-zero probability so it never zeroes out the product.
Variants
The three common Naive Bayes variants differ only in how they model the likelihood P(xᵢ | y) — that is, in their assumption about how features are distributed. Choose the one whose assumption fits your features.
| Variant | Feature type | Likelihood P(xᵢ | y) | Typical use |
|---|---|---|---|
| Multinomial | Word counts / frequencies | (count+α) / (N_c+α|V|) | Text & document classification (how often words appear) |
| Bernoulli | Binary present / absent | pᵢ if present, (1−pᵢ) if absent | Short text where word presence matters; explicitly penalizes absent words |
| Gaussian | Continuous real-valued | N(xᵢ; μᵧ, σᵧ²) | Numeric features (height, sensor readings); fits a per-class mean μ and variance σ² |
Key contrast: Multinomial counts how many times each word occurs, while Bernoulli only records whether a word is present and actively factors in the probability of words that are absent. Gaussian drops counting entirely and models each continuous feature with a bell curve per class, plugging xᵢ into the normal density N(xᵢ; μ, σ²) = (1/√(2πσ²))·e^(−(xᵢ−μ)²/(2σ²)).
Why It Works Despite Being “Wrong”
The independence assumption is clearly false for language, yet Naive Bayes is a strong text/spam classifier. Why?
We only need the argmax
Classification needs the ranking of classes to be right, not the probabilities themselves. Correlated features make the posterior probabilities poorly calibrated (often pushed toward 0 or 1), but the winning class is frequently still correct. Good decisions survive bad probability estimates.
Robust counts, low variance
Estimating each P(xᵢ|y) separately needs little data and has low variance, so the model rarely overfits. Many weak, slightly redundant word-cues accumulate via the sum of log-probabilities into a confident, accurate verdict — exactly the regime spam filtering lives in.
Pros and Cons
| Pros | Cons |
|---|---|
| Extremely fast to train and predict (just counting) | Independence assumption ignores feature correlations |
| Works well with little training data | Posterior probabilities are often poorly calibrated |
| Scales to very high-dimensional, sparse text features | Needs smoothing to handle unseen words |
| Simple, interpretable, strong baseline | Often beaten by discriminative models when data is plentiful |
Key Takeaways
- Naive Bayes applies Bayes' theorem: P(y|x) ∝ P(x|y)·P(y), dropping the constant evidence P(x).
- The “naive” conditional-independence assumption factorizes the likelihood into ∏ᵢ P(xᵢ|y), making estimation a matter of counting.
- Prediction is the MAP rule ŷ = argmax_y P(y)·∏ᵢ P(xᵢ|y), computed in log space (log P(y) + Σ log P(xᵢ|y)) to avoid numerical underflow.
- Laplace/additive smoothing (add α, with α = 1 the classic case) cures the zero-frequency problem so an unseen word cannot zero out a class.
- Variants — Multinomial (counts), Bernoulli (present/absent), Gaussian (continuous) — differ only in how they model P(xᵢ|y).
- It stays a strong text/spam classifier despite false independence because correct ranking, not calibrated probability, is what classification needs.