Principal Component Analysis (PCA)
Introduction
Principal Component Analysis (PCA) is the workhorse of linear dimensionality reduction. Given high-dimensional data, it finds a new set of axes — the principal components — ordered so that the first captures as much of the data's variance as possible, the second captures as much of the remaining variance as possible (while being perpendicular to the first), and so on. Keeping only the first few of these axes lets you compress the data, visualize it, remove noise, or speed up downstream models, while losing as little information as the geometry allows.
PCA is unsupervised (it never looks at labels) and linear (every new axis is a fixed linear combination of the original features). Its power comes from a single observation: the directions of maximum variance in centered data are exactly the eigenvectors of the covariance matrix.
The one-sentence version: PCA rotates your coordinate system to align with the directions in which the data varies most, then lets you drop the directions that barely vary.
How it works, step by step
1. Center the data (subtract the mean)
Compute the mean of each feature, μ = (1/n) Σᵢ xᵢ, and subtract it from every point: x̃ᵢ = xᵢ − μ. Centering is not optional — “direction of maximum variance” is only meaningful relative to the data's center. Skip it and the first component gets dragged toward the origin instead of describing the spread of the cloud.
2. Build the covariance matrix
With the centered data stacked into an n×d matrix X̃, the d×d sample covariance matrix is
C = (1 / (n − 1)) · X̃ᵀ X̃
Entry Cⱼₖ measures how features j and k vary together; the diagonal entries are the per-feature variances. C is symmetric and positive semi-definite, which guarantees real, non-negative eigenvalues and orthogonal eigenvectors.
3. Eigen-decompose: components and their variances
Solve C vᵢ = λᵢ vᵢ. The eigenvectors vᵢ are the principal components (the new axes), and the eigenvalue λᵢ is the variance of the data along vᵢ. Sort them so λ₁ ≥ λ₂ ≥ … ≥ λ_d. Because C is symmetric, the eigenvectors are mutually orthogonal — the components are perpendicular by construction.
This is the key fact: maximizing variance subject to a unit-length, orthogonal-to-previous constraint is exactly the eigenvalue problem for C.
4. Project onto the top-k components
Stack the top k eigenvectors as columns of a d×k matrix W = [v₁ … v_k]. The reduced representation of a centered point is its coordinates in this new basis:
z = Wᵀ x̃ (k numbers instead of d)
To get an approximate reconstruction back in the original space: x̃ ≈ W z = W Wᵀ x̃. The information you lose is precisely the variance along the discarded components, Σ for i > k of λᵢ.
Proportion of variance explained
Each component accounts for a share of the total variance equal to its eigenvalue divided by the sum of all eigenvalues:
explained(i) = λᵢ / (λ₁ + λ₂ + … + λ_d)
A common rule of thumb is to choose k as the smallest number of components whose cumulative explained variance reaches a target like 90% or 95%.
Interactive: 2D PCA visualizer
The widget computes the real 2×2 covariance matrix from the (centered) point cloud and its exact eigenvectors and eigenvalues in closed form. Reshape and rotate the cloud with the sliders and watch PC1 and PC2 track the directions of maximum variance. Toggle the projection to collapse the data onto PC1 and see precisely what dimensionality reduction discards.
The link to SVD
In practice PCA is usually computed with the singular value decomposition (SVD) of the centered data matrix rather than by forming the covariance matrix explicitly. Writing the centered data as X̃ = U Σ Vᵀ:
- The right singular vectors (columns of V) are the principal components — the same eigenvectors of C.
- The squared singular values relate to the eigenvalues by λᵢ = σᵢ² / (n − 1).
- The projected coordinates (scores) are UΣ, equivalently X̃V.
Why prefer SVD? Forming C = X̃ᵀX̃ squares the condition number and can lose numerical precision. Running SVD directly on X̃ is more numerically stable and avoids ever materializing the d×d covariance matrix — a real advantage when d is large.
To standardize or not?
PCA is driven by variance, so it is sensitive to the scale of your features. If one feature is measured in millimeters and another in kilometers, the millimeter feature will have a huge numerical variance and dominate the first component for reasons that have nothing to do with the underlying structure.
The fix is to standardize each feature to unit variance (subtract the mean, divide by the standard deviation) before running PCA. This is equivalent to running PCA on the correlation matrix instead of the covariance matrix.
| Situation | What to do |
|---|---|
| Features are on different units / wildly different scales | Standardize first (use the correlation matrix) |
| Features share a meaningful common unit (e.g. all pixel intensities) | Centering alone is often appropriate (covariance matrix) |
| Always | Center (subtract the mean) — this is mandatory, not a choice |
Strengths, limits, and pitfalls
Strengths
- • Fast, deterministic, and has a closed-form solution
- • Decorrelates features and compresses redundant ones
- • Great for visualization (project to 2–3 components)
- • Can denoise by dropping low-variance directions
Limits & pitfalls
- • Linear only — cannot capture curved structure (use kernel PCA / t-SNE / UMAP)
- • Unsupervised — high-variance directions are not always the discriminative ones
- • Components are combinations of all features, so they can be hard to interpret
- • Forgetting to standardize lets large-scale features dominate
- • Assumes variance equals importance, which is not always true
Key Takeaways
- PCA finds orthogonal directions of maximum variance; centering the data first is mandatory, not optional.
- The principal components are the eigenvectors of the covariance matrix; the eigenvalues are the variance captured along each component.
- Dimensionality reduction = projecting centered data onto the top-k components; what you lose is the variance along the discarded ones.
- Proportion of variance explained by component i is λᵢ / Σλ — use the cumulative curve to pick k.
- PCA is most stably computed via the SVD of the centered data matrix, where right singular vectors are the components and λᵢ = σᵢ²/(n−1).
- PCA is linear and unsupervised; standardize features when they are on different scales, and reach for nonlinear methods when the structure is curved.