Cross-Validation
Why a single split is not enough
When we train a model we ultimately care about one thing: how well will it perform on data it has never seen? The simplest way to estimate this is a single train/test split — hold out, say, 20% of the data, train on the rest, and score on the held-out portion.
The problem is that this single number is noisy. Which examples happened to land in the test set is partly luck. A different random split can swing the reported accuracy by several percentage points, especially on small datasets. You might get an unlucky split and reject a perfectly good model, or a lucky split and ship a model that is worse than you think.
The core idea of cross-validation: instead of trusting one split, make several, and average the results. Averaging many estimates of the same quantity reduces the variance of that estimate — the average is a more stable, trustworthy measure of how the model generalizes.
k-Fold Cross-Validation
The workhorse procedure is k-fold cross-validation:
- Split the data into k roughly equal-sized groups called folds (common choices: k = 5 or k = 10).
- For each fold in turn, hold it out as the validation set and train on the other k−1 folds.
- Score the trained model on the held-out fold. This gives you k separate scores.
- Average the k scores to get the cross-validation estimate (and report the spread / standard deviation alongside it).
Because the folds rotate, every example is used for validation exactly once and for training exactly k−1 times. No data is wasted, and the averaged score is far more stable than any single split. You do pay a cost: you train the model k times instead of once.
CV score = (1/k) · Σᵢ scoreᵢ where scoreᵢ is the metric on the i-th held-out fold. The reported uncertainty is usually the standard deviation of those k scores.
Interactive: k-Fold Visualizer
Below is a row of data items. Pick the number of folds, then step through each round to watch which block is held out for validation (orange) and which is used for training (green). Read off the score for each round and the averaged CV estimate with its spread.
Validation score per round
Averaged CV estimate
Important variants
Stratified k-Fold
For classification, a plain split can accidentally put very few (or zero) examples of a rare class into a fold, making its score meaningless. Stratified k-fold splits each class separately so that every fold preserves the overall class proportions. It is the sensible default for classification problems, especially with imbalanced classes. Toggle "Stratified" in the demo above to see the class mix held constant across folds.
Leave-One-Out (LOOCV): k = n
Take k all the way to the number of examples n. Each fold is a single example: train on n−1 points, validate on the one left out, repeat n times. LOOCV uses almost all the data for every training run, so its estimate has low bias. But it is expensive (n model fits) and the per-fold scores are highly correlated, which can give the estimate high variance. For most problems 5- or 10-fold strikes a better balance.
Other useful variants
- Repeated k-fold: run k-fold several times with different shuffles and average — further reduces estimate variance.
- Group k-fold: when rows share a group (e.g. multiple samples from one patient), keep a whole group entirely in train or validation so it cannot leak across the split.
- Time-series CV: for ordered data, never train on the future. Use expanding/rolling windows where validation always comes after training in time.
Train / Validation / Test — keep the test set sacred
A common confusion is between the validation set (used to choose the model) and the test set (used to report the final, honest performance). The cleanest setup is a three-way split:
| Set | Purpose | Touched how often? |
|---|---|---|
| Training | Fit model parameters (weights). | Every training run. |
| Validation | Tune hyperparameters and choose between models. This is exactly what cross-validation gives you (using the training data itself). | Many times, during tuning. |
| Test | Final unbiased estimate of generalization, reported once. | Exactly once, at the very end. |
Why the test set must stay untouched (data leakage): every time you look at a set and use it to make a decision, you tune toward it. If you pick hyperparameters using the test set, your test score becomes optimistically biased — it no longer reflects truly unseen data. The test set is spent the moment it influences a choice. Do all tuning with cross-validation on the training data, then evaluate on the test set once and do not go back and tweak.
Using CV for tuning and model comparison
Cross-validation turns "is this model good?" into a number you can optimize. The standard recipes:
- Hyperparameter selection: for each candidate setting (e.g. each regularization strength λ, or each k in KNN), run k-fold CV and keep the setting with the best average score. This is exactly what a grid search or random search does internally.
- Model comparison: run the same folds for each candidate model and compare their mean CV scores; the spread tells you whether a difference is real or within the noise.
The subtle leak in tuning: once you have selected hyperparameters by their CV score, that CV score is itself optimistically biased — you chose the winner because it scored high on those folds. To report an honest number for the tuned pipeline, you need a separate held-out test set, or nested cross-validation.
Nested cross-validation (briefly)
Nested CV gives an unbiased estimate of a tuned model's performance when you have no separate test set. It uses two loops:
- Inner loop: on each outer training portion, run ordinary k-fold CV to select the best hyperparameters.
- Outer loop: retrain with those chosen hyperparameters and evaluate on the outer held-out fold — which the inner loop never saw.
Because hyperparameter selection happens entirely inside the inner loop, the outer scores measure the whole tune-and-fit procedure on genuinely unseen data. The cost is high (roughly k_outer × k_inner × (#settings) fits), so it is used when an honest estimate matters and data is scarce.
How k affects the estimate (bias–variance)
Choosing k is itself a bias–variance tradeoff — not of the model, but of the CV estimate of its performance:
| k | Train size per fold | Bias of estimate | Variance & cost |
|---|---|---|---|
| Small (e.g. 2–3) | Smaller — each model trains on much less data. | Higher (pessimistic): trains on less data than the final model will, so it underestimates true performance. | Lower cost; folds are fairly independent. |
| Large (e.g. 10) | Larger — close to the full training set. | Lower (more optimistic and realistic). | Higher cost; more fits. |
| k = n (LOOCV) | n − 1 (almost everything). | Lowest bias. | Highest cost; correlated folds can give high variance. |
The usual sweet spot is k = 5 or 10: enough training data per fold to keep bias small, enough folds to average down the variance, and a manageable number of training runs.
Common pitfalls
- Preprocessing leakage: fit scalers, feature selection, or imputation inside each fold (on training data only). Fitting them on the whole dataset before splitting leaks information from the validation folds.
- Tuning on the test set: using the test set to pick hyperparameters destroys its honesty — that is what validation/CV is for.
- Ignoring structure: with grouped or time-ordered data, random k-fold leaks; use group or time-series CV instead.
- Forgetting stratification: on imbalanced classification, non-stratified folds can produce wildly varying or degenerate fold scores.
Key Takeaways
- A single train/test split gives a noisy, lucky-or-unlucky estimate of generalization; cross-validation averages over several splits to make it stable.
- k-fold CV splits data into k folds, trains on k−1 and validates on the held-out fold, rotating so every point is validated exactly once, then averages the k scores.
- Stratified k-fold preserves class proportions in every fold — the default for classification.
- Leave-one-out (k = n) has low bias but is expensive and can have high variance; k = 5 or 10 is the usual sweet spot.
- Keep the test set untouched during tuning — any decision made from it leaks information and inflates your reported score.
- Use CV for hyperparameter selection and model comparison; use nested CV (or a held-out test set) to get an honest estimate of the tuned pipeline.