Covariance matrix
Also: covariance matrices
For d features, the d×d symmetric matrix C with Cᵢⱼ = Cov(xᵢ, xⱼ). It describes the shape and orientation of a data cloud; its eigenvectors are the principal components.
C = (1 / (n − 1)) X̃ᵀ X̃ (X̃ = centered data)
Example
For the points (1, 2), (2, 4), (3, 6): Var(x) = 1, Var(y) = 4, Cov(x, y) = 2, so C = [[1, 2], [2, 4]]. Its top eigenvector points along (1, 2), the line the data lies on.