2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread

Why is the equivalence between minimizing reconstruction error and maximizing projected variance considered one of the most profound insights in applied mathematics? Because it bridges two completely different philosophies of data analysis: lossy compression (how accurately can we reconstruct our inputs?) and information theory (how well can we distinguish different observations from one another along a low-dimensional summary?).

In this Article

• 1. From Point-Wise Pythagoras to Dataset-Wide Optimization
• 2. Why Maximizing Variance Preserves Pairwise Distinguishability
• 3. The Signal-to-Noise Interpretation of Principal Axes
• 4. Worked Numerical Trade-Off Calculation
• 5. Active Recall Check and Key Takeaways

1. From Point-Wise Pythagoras to Dataset-Wide Optimization

In Microlearning 2.2, we proved that for every individual centered data point $\mathbf{x}_i$ and any unit vector $\mathbf{u}$:

$$e_i^2(\mathbf{u}) = \|\mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}\|^2 = \|\mathbf{x}_i\|^2 - (\mathbf{u}^T\mathbf{x}_i)^2$$

Summing over all $i = 1, \dots, n$ and dividing by $n$ gives the dataset-level objective function:

$$\underbrace{\frac{1}{n}\sum_{i=1}^n e_i^2(\mathbf{u})}_{\text{MSE}(\mathbf{u})} = \underbrace{\frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i\|^2}_{E_{\text{total}} \text{ (Constant)}} - \underbrace{\frac{1}{n}\sum_{i=1}^n (\mathbf{u}^T\mathbf{x}_i)^2}_{V(\mathbf{u}) \text{ (Projected Variance)}}$$

In optimization theory, minimizing $\text{Constant} - V(\mathbf{u})$ over $\mathbf{u}$ has the exact same set of optimal solutions $\mathbf{u}^*$ as maximizing $V(\mathbf{u})$:

$$\underset{\|\mathbf{u}\|=1}{\arg\min} \, \text{MSE}(\mathbf{u}) = \underset{\|\mathbf{u}\|=1}{\arg\max} \, V(\mathbf{u})$$

2. Why Maximizing Variance Preserves Pairwise Distinguishability

There is an even deeper geometric way to view this duality: what happens to the distance between any two shadows $\hat{\mathbf{x}}_i$ and $\hat{\mathbf{x}}_k$ on the line? By the Pythagorean theorem applied to the pairwise difference vector $\mathbf{x}_i - \mathbf{x}_k$:

$$\underbrace{\|\mathbf{x}_i - \mathbf{x}_k\|^2}_{\text{True 2D Distance}^2} = \underbrace{(z_i - z_k)^2}_{\text{1D Shadow Distance}^2} + \underbrace{\|\mathbf{r}_i - \mathbf{r}_k\|^2}_{\text{Residual Difference}^2}$$

Because $\|\mathbf{r}_i - \mathbf{r}_k\|^2 \ge 0$, projecting onto a line can only shrink pairwise distances: $(z_i - z_k)^2 \le \|\mathbf{x}_i - \mathbf{x}_k\|^2$. If you pick a bad axis, distinct data points collapse on top of each other's shadows! Maximizing the variance $V(\mathbf{u}) = \frac{1}{2n^2}\sum_{i=1}^n \sum_{k=1}^n (z_i - z_k)^2$ prevents shadow collisions by keeping the projected shadows as far apart—and as faithful to their true 2D distances—as possible.

   BAD AXIS (Low Variance / High Error):         GOOD AXIS (Max Variance / Min Error):
   Shadows collide! Points become indistinguishable.  Shadows stay spread out and distinct!

          * x_1                                                    * x_3
          |                                                  * x_2  \
          |    * x_2                                   * x_1  \      (o)
          |    |    * x_3                               \     (o)    /
          v    v    v                                   (o)   /     /
   ------(o)--(o)--(o)------> u_bad                     /    /     /  <-- u_optimal
      (Crushed together!)                            (Well-separated shadows!)

3. The Signal-to-Noise Interpretation of Principal Axes

4. Worked Numerical Trade-Off Calculation

Suppose a centered 5D dataset has total scatter $E_{\text{total}} = 50.0$. We test three candidate unit vectors:

Candidate Unit Vector	Measured Shadow Variance $V(\mathbf{u})$	Implied Reconstruction Error $\text{MSE}(\mathbf{u}) = 50 - V(\mathbf{u})$	Percentage of Total Information Retained
Direction $\mathbf{u}_A$	$$12.5$$	$$37.5$$	$$25\%$$
Direction $\mathbf{u}_B$	$$35.0$$	$$15.0$$	$$70\%$$
Optimal PC1 $\mathbf{u}^*$	$$42.5$$	$$7.5$$	$$85\%$$

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: Why can the 1D shadow distance $|z_i - z_k|$ between two observations never exceed their true Euclidean distance $\|\mathbf{x}_i - \mathbf{x}_k\|$?

Answer: By the Cauchy-Schwarz inequality (or the Pythagorean identity on $\mathbf{x}_i - \mathbf{x}_k$), $|z_i - z_k| = |\mathbf{u}^T(\mathbf{x}_i - \mathbf{x}_k)| \le \|\mathbf{u}\| \, \|\mathbf{x}_i - \mathbf{x}_k\| = \|\mathbf{x}_i - \mathbf{x}_k\|$ since $\|\mathbf{u}\| = 1$. Maximizing variance chooses $\mathbf{u}$ to make $|z_i - z_k|^2$ as close to the upper bound $\|\mathbf{x}_i - \mathbf{x}_k\|^2$ as possible on average.

• Equivalence of Objectives: $\arg\min \text{MSE}(\mathbf{u}) = \arg\max V(\mathbf{u})$ because their sum is fixed at $E_{\text{total}}$.
• Pairwise Distance Preservation: Maximizing variance maximizes the average squared distance between projected points, preventing distinct observations from collapsing onto the same shadow.
• Canonical References: See Shlens' PCA Tutorial on Signal-to-Noise Ratio and Variance.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 2 Overview: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance
• Microlearning 2.1: 2.1 Total Data Energy as an Invariant Sum of Squared Norms
• Microlearning 2.2: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error
• Microlearning 2.3: 2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread
• Previous Lesson: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error
• Next Lesson: Chapter 3: Visualizing Linear Transformations, Eigenvectors, and Eigenvalues Without Memorization