Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance

When textbooks introduce Principal Component Analysis, they present two seemingly unrelated definitions: Karl Pearson's (1901) formulation asks for the subspace that minimizes orthogonal reconstruction error, whereas Harold Hotelling's (1933) formulation asks for the subspace that maximizes the variance of the projected points.

Why does minimizing error automatically maximize variance? The bridge between these two worlds requires no calculus or matrix determinants—only the Pythagorean theorem applied to the right triangle formed by each data point, its shadow on the projection line, and its perpendicular residual.

In this Article

• 1. The Two Historical Faces of PCA: Pearson (1901) vs. Hotelling (1933)
• 2. The Right Triangle Inside Every Orthogonal Projection
• 3. Summing Over the Dataset: The Conservation Law of Total Scatter
• 4. Step-by-Step Numerical Proof Across Multiple Angles
• 5. The Rotating Flashlight Analogy and Its Limits
• 6. Critical Misconceptions and Edge Cases
• 7. Active Recall Check and Key Takeaways

1. The Two Historical Faces of PCA: Pearson (1901) vs. Hotelling (1933)

Suppose you have $n$ centered observations $\mathbf{x}_1, \dots, \mathbf{x}_n \in \mathbb{R}^d$ and a candidate unit vector $\mathbf{u}$ ($\|\mathbf{u}\|=1$). Consider two goals:

• Goal A (Compression / Minimal Distortion): Make the perpendicular error vectors $\mathbf{r}_i = \mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$ as short as possible on average:

  $$\min_{\|\mathbf{u}\|=1} \frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}\|^2$$

• Goal B (Information Preservation / Maximal Spread): Make the projected 1D scores $z_i = \mathbf{u}^T\mathbf{x}_i$ as spread out from their zero mean as possible:

  $$\max_{\|\mathbf{u}\|=1} \frac{1}{n}\sum_{i=1}^n (\mathbf{u}^T\mathbf{x}_i)^2$$

2. The Right Triangle Inside Every Orthogonal Projection

For any single point $\mathbf{x}_i$, look at the triangle with vertices at the origin $\mathbf{0}$, the projected shadow $\hat{\mathbf{x}}_i = (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$, and the original point $\mathbf{x}_i$:

                             * x_i (Data Point)
                            /|
                           / |
     Hypotenuse ||x_i||   /  |  Opposite Side: Error e_i = ||x_i - x_hat_i||
     (FIXED by the data! /   |  (Shrinks as u aligns with the cloud)
      Never changes as  /    |
      we rotate u!)    /   _ |
                      /   | ||  <-- 90-degree Right Angle
            Origin (0)=======(o)---------> Candidate Unit Axis u
                      Base: Shadow Length |z_i| = |u^T x_i|
                      (Grows as u aligns with the cloud)

Because the residual vector $\mathbf{r}_i = \mathbf{x}_i - \hat{\mathbf{x}}_i$ is perpendicular to the axis $\mathbf{u}$, the Pythagorean theorem holds for every single point $i$:

$$\underbrace{\|\mathbf{x}_i\|^2}_{\text{Hypotenuse}^2} = \underbrace{(\mathbf{u}^T\mathbf{x}_i)^2}_{\text{Base (Shadow)}^2} + \underbrace{\|\mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}\|^2}_{\text{Height (Error)}^2}$$

3. Summing Over the Dataset: The Conservation Law of Total Scatter

Now average this Pythagorean equation across all $n$ points in the centered dataset:

Look at the left-hand side: $E_{\text{total}} = \frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i\|^2$ does not contain $\mathbf{u}$ at all! As you spin the line $\mathbf{u}$ around the origin, the data points stay frozen in space, so their distances from the origin never change.

Because $V(\mathbf{u}) + \text{MSE}(\mathbf{u}) = E_{\text{total}} = \text{constant}$, we can write:

$$\text{MSE}(\mathbf{u}) = E_{\text{total}} - V(\mathbf{u})$$

Minimizing $\text{MSE}(\mathbf{u})$ is therefore algebraically identical to maximizing $V(\mathbf{u})$! Making the perpendicular errors as small as possible forces the projected shadows on the line to spread out as widely as possible.

4. Step-by-Step Numerical Proof Across Multiple Angles

Take our 4-point centered dataset from Chapter 1: $\mathbf{x}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}, \mathbf{x}_2 = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, \mathbf{x}_3 = \begin{bmatrix} -1 \\ -2 \end{bmatrix}, \mathbf{x}_4 = \begin{bmatrix} -2 \\ -1 \end{bmatrix}$. Each point has $\|\mathbf{x}_i\|^2 = 2^2 + 1^2 = 5$, so $E_{\text{total}} = 5.0$.

Candidate Axis Angle	Unit Vector $\mathbf{u}$	Projected Variance $V(\mathbf{u})$	Reconstruction Error $\text{MSE}(\mathbf{u})$	Sum $V(\mathbf{u}) + \text{MSE}(\mathbf{u})$
$0^\circ$ (Horizontal $x_1$)	$$\begin{bmatrix} 1 & 0 \end{bmatrix}^T$$	$$\frac{4+1+1+4}{4} = 2.50$$	$$\frac{1+4+4+1}{4} = 2.50$$	$$2.50 + 2.50 = 5.00$$
$45^\circ$ (Optimal PC1)	$$\begin{bmatrix} 1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix}^T$$	$$\frac{4.5+4.5+4.5+4.5}{4} = 4.50$$	$$\frac{0.5+0.5+0.5+0.5}{4} = 0.50$$	$$4.50 + 0.50 = 5.00$$
$-45^\circ$ (Worst Axis PC2)	$$\begin{bmatrix} -1/\sqrt{2} & 1/\sqrt{2} \end{bmatrix}^T$$	$$\frac{0.5+0.5+0.5+0.5}{4} = 0.50$$	$$\frac{4.5+4.5+4.5+4.5}{4} = 4.50$$	$$0.50 + 4.50 = 5.00$$

5. The Rotating Flashlight Analogy and Its Limits

Imagine holding a rigid wooden rod of length $L = 5$ feet hinged at the origin in a dark room. If you shine a light straight down onto the floor, the shadow on the floor has length $L\cos\theta$ and the vertical height off the floor is $L\sin\theta$. Tilting the rod closer to the floor lengthens the shadow while shrinking the height, yet $(\text{shadow})^2 + (\text{height})^2$ is forever locked at $L^2 = 25$.

Where the analogy breaks down: In PCA, the data points themselves do not tilt; the projection screen $\mathbf{u}$ rotates around a fixed constellation of $n$ points simultaneously, so the identity holds as an ensemble average across all $n$ right triangles at once.

6. Critical Misconceptions and Edge Cases

7. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: A centered 3D dataset has feature variances $\text{Var}(x_1) = 10$, $\text{Var}(x_2) = 6$, and $\text{Var}(x_3) = 4$. If the first principal component PC1 captures a projected variance of $\lambda_1 = 15$, what is the exact mean squared reconstruction error when compressing the 3D points onto the 1D line PC1?

Answer: First, the total data energy is the sum of the feature variances: $E_{\text{total}} = 10 + 6 + 4 = 20$. By the Pythagorean conservation law, $\text{MSE}(\mathbf{u}_1) = E_{\text{total}} - V(\mathbf{u}_1) = 20 - 15 = 5$. No individual point residuals need to be computed!

• Pythagorean Duality: Every orthogonal projection forms a right triangle with constant hypotenuse $\|\mathbf{x}_i\|^2$.
• Zero-Sum Trade-off: $E_{\text{total}} = V(\mathbf{u}) + \text{MSE}(\mathbf{u})$, so minimizing reconstruction error is identical to maximizing projected variance.
• Canonical References: Read CrossValidated's Highest-Voted Geometric Breakdown of PCA and An Introduction to Statistical Learning (Chapter 12: Unsupervised Learning).

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 2 Overview: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance
• Microlearning 2.1: 2.1 Total Data Energy as an Invariant Sum of Squared Norms
• Microlearning 2.2: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error
• Microlearning 2.3: 2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread
• Previous Lesson: 1.3 Orthogonal Reconstruction Error vs. Vertical Regression Residuals
• Next Lesson: 2.1 Total Data Energy as an Invariant Sum of Squared Norms