Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up)

A complete 6-chapter, 24-article course uniting the Top-Down optimization view of Principal Component Analysis (minimizing orthogonal reconstruction error via the Pythagorean theorem) with the Bottom-Up geometric view of linear algebra (matrices as space transformations, eigenvectors as invariant axes, covariance ellipses, Lagrange multipliers, and SVD).

Public

24 Articles

Chapter 1: The Geometry of Data Compression — Centering, Projections, and Orthogonal Shadows

Learn the top-down foundation of Principal Component Analysis (PCA): how mean-centering shifts a dataset to the origin, how dot-product projections cast 1D shadows, and why PCA minimizes perpendicular rather than vertical errors.

Linear Algebra & Data Science

1.1 Centering Multivariate Data and Shifting the Origin to the Mean

Calculate sample mean vectors, translate multivariate datasets so their centroid sits at the origin, and diagnose how uncentered data distorts subspace projections.

Linear Algebra & Data Science

1.2 Unit Direction Vectors and Scalar Dot-Product Projections

Normalize direction vectors to unit length, compute 1D compressed scores via dot products, and reconstruct 2D/3D shadow coordinates along a candidate axis.

Linear Algebra & Data Science

1.3 Orthogonal Reconstruction Error vs. Vertical Regression Residuals

Distinguish Principal Component Analysis (orthogonal total least squares) from Ordinary Least Squares (vertical regression) and prove why symmetry across features demands perpendicular residuals.

Linear Algebra & Data Science

Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance

Discover the geometric heart of PCA: how the right triangle between a data point, its shadow, and its orthogonal residual proves that minimizing compression error is mathematically identical to maximizing variance.

Linear Algebra & Data Science

2.1 Total Data Energy as an Invariant Sum of Squared Norms

Calculate the total scatter (average squared Euclidean norm) of a centered dataset and prove why rotating your coordinate axes never changes total data energy.

Linear Algebra & Data Science

2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error

Prove algebraically and geometrically why the projection vector and residual vector are orthogonal, and derive the exact right-triangle identity for every observation.

Linear Algebra & Data Science

2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread

Average the Pythagorean identity across an entire dataset to prove the conservation law of PCA: every unit of variance captured along the axis is a unit of reconstruction error eliminated.

Linear Algebra & Data Science

Chapter 3: Visualizing Linear Transformations, Eigenvectors, and Eigenvalues Without Memorization

Build bottom-up geometric intuition for linear algebra: see matrices as space-warping machines, eigenvectors as stationary axes of pure stretch, and determinants as volume scaling factors.

Linear Algebra & Data Science

3.1 Matrices as Geometric Grid Transformations: Stretching, Shearing, and Rotating

Read any matrix by looking at where its columns land: trace how basis vectors transform unit squares into parallelograms and unit circles into tilted ellipses.

Linear Algebra & Data Science

3.2 Eigenvectors and Eigenvalues as Invariant Axes of Pure Stretch

Understand Av = λv from first principles: why almost every vector gets knocked off its span during a matrix transformation, while eigenvectors stay locked on their line of action.

Linear Algebra & Data Science

3.3 Why det(A - λI) = 0 Finds Non-Zero Vectors in the Collapse Nullspace

Stop memorizing the characteristic polynomial as a magic recipe: see why subtracting λI shifts the transformation until space squishes flat into a lower dimension.

Linear Algebra & Data Science

Chapter 4: What a Covariance Matrix Really Is — From Scatter Clouds to Orthogonal Stretch Axes

Connect statistics to linear algebra: see how C = (1/n)XᵀX encodes all pairwise feature relationships, how uᵀCu measures directional variance, and why symmetric matrices always have perpendicular eigenvectors.

Linear Algebra & Data Science

4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data

Understand why multiplying a centered data matrix by its transpose (XᵀX) automatically computes every feature's variance on the diagonal and every pairwise covariance off the diagonal.

Linear Algebra & Data Science

4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction

Derive the bridge formula V(u) = uᵀCu showing how the covariance matrix acts as a directional variance meter for any unit vector u in space.

Linear Algebra & Data Science

4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors

Prove in three clean lines of algebra why symmetric matrices never shear space obliquely—guaranteeing real eigenvalues and mutually perpendicular principal components.

Linear Algebra & Data Science

Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation

Witness the exact moment Top-Down optimization meets Bottom-Up linear algebra: how maximizing uᵀCu on the unit sphere uᵀu = 1 forces gradient alignment 2Cu = λ(2u) and proves that eigenvalue λ is the projected variance.

Linear Algebra & Data Science

5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential

See why maximizing uᵀCu without a length constraint is an ill-posed problem that explodes to infinity, and how restricting u to the unit sphere isolates pure direction from ruler scale.

Linear Algebra & Data Science

5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically

Visualize Lagrange multipliers without black-box calculus: see why the highest variance contour must be tangent to the unit circle, forcing their perpendicular gradients 2Cu and 2u to align.

Linear Algebra & Data Science

5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u

Substitute Cu = λu into V(u) = uᵀCu to prove that the Lagrange multiplier λ is not a mere mathematical artifact—it is literally the variance of the data along that eigenvector.

Linear Algebra & Data Science

Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD

Master the complete practical execution of PCA: work through a 2D dataset by hand, verify trace and error conservation, choose k components via explained variance ratios, and understand why production libraries use SVD.

Linear Algebra & Data Science

6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset

Walk through every arithmetic step of PCA on a concrete 4-point dataset: centering, forming the 2×2 covariance matrix, solving the characteristic quadratic, projecting scores, and verifying Pythagorean error.

Linear Algebra & Data Science

6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components

Use the matrix trace identity Trace(C) = Σλᵢ to compute explained variance ratios, read scree plots, and understand why feature standardization (z-scores) is critical when units differ.

Linear Algebra & Data Science

6.3 Why Production PCA Uses Singular Value Decomposition (SVD) Instead of Forming XᵀX

Discover how X = UΣVᵀ connects singular values σᵢ to eigenvalues λᵢ = σᵢ²/n, and why scikit-learn and NumPy compute PCA via SVD to prevent condition-number squaring and floating-point precision loss.

Linear Algebra & Data Science