Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation

We have arrived at the intellectual climax of the course. In Chapters 1 and 2, we started with a purely practical goal—compressing data with the smallest possible orthogonal reconstruction error—and proved via the Pythagorean theorem that it is equivalent to maximizing projected variance $\mathbf{u}^T C \mathbf{u}$ over all unit vectors $\mathbf{u}^T \mathbf{u} = 1$. In Chapters 3 and 4, we explored eigenvectors $C\mathbf{u} = \lambda\mathbf{u}$ as axes of pure geometric stretch.

Now we ask the question that unites both paths: Why does maximizing variance $\mathbf{u}^T C \mathbf{u}$ subject to $\mathbf{u}^T \mathbf{u} = 1$ produce the exact eigenvalue equation $C\mathbf{u} = \lambda\mathbf{u}$? Even more remarkably, why does the Lagrange multiplier $\lambda$ introduced solely to enforce the unit-length constraint turn out to equal the exact variance along $\mathbf{u}$?

In this Article

• 1. Formulating the Constrained Optimization Problem
• 2. Geometric Proof: Kissing Contours and Parallel Gradients
• 3. Calculus Proof: Differentiating the Lagrangian L(u, λ)
• 4. The Payoff: Proving uᵀCu = λ and Ranking Principal Components
• 5. Deflation: Why the 2nd Principal Component Is the 2nd Eigenvector
• 6. Complete Worked Numerical Example
• 7. Active Recall Check and Key Takeaways

1. Formulating the Constrained Optimization Problem

Why can we not simply maximize $f(\mathbf{u}) = \mathbf{u}^T C \mathbf{u}$ without constraints? Because $\mathbf{u}^T C \mathbf{u}$ is quadratic in the length of $\mathbf{u}$: if you replace $\mathbf{u}$ with $10\mathbf{u}$, the value of $(10\mathbf{u})^T C (10\mathbf{u}) = 100(\mathbf{u}^T C \mathbf{u})$ explodes by $100\times$ without rotating the line at all! To optimize direction rather than vector length, we must lock $\mathbf{u}$ onto the unit hypersphere $g(\mathbf{u}) = \mathbf{u}^T \mathbf{u} = 1$:

$$\max_{\mathbf{u} \in \mathbb{R}^d} \; \mathbf{u}^T C \mathbf{u} \quad \text{subject to} \quad \mathbf{u}^T \mathbf{u} = 1$$

2. Geometric Proof: Kissing Contours and Parallel Gradients

Imagine walking around the unit circle $g(\mathbf{u}) = \mathbf{u}^T \mathbf{u} = 1$ while reading the height of the variance function $f(\mathbf{u}) = \mathbf{u}^T C \mathbf{u}$. If the level curve of $f(\mathbf{u})$ crosses the unit circle at an angle, you can slide along the circle in the uphill direction to reach a higher variance.

                     u2 ^
                        |        . - ~ ~ - .   <-- Variance contour u^T C u = lambda_1
                        |    . '      *      ' .
                   +----+--'---------/(*)-------'--+
                  /     |/          /   \           \   Grad f = 2 C u  and  Grad g = 2 u
                 |      0          0     \           |  point in the EXACT SAME direction!
                  \    /|                 \         /   Therefore: C u = lambda u !
                   +--/-+------------------+-------+
                        |    Unit Circle u^T u = 1

At the exact point of maximum variance, the elliptical contour of $f(\mathbf{u}) = \mathbf{u}^T C \mathbf{u}$ must kiss (be tangent to) the unit circle $g(\mathbf{u}) = \mathbf{u}^T \mathbf{u} = 1$. Because two tangent curves share the same normal line, their gradient vectors must be parallel:

$$\nabla f(\mathbf{u}) = \lambda \nabla g(\mathbf{u}) \implies 2 C \mathbf{u} = \lambda (2 \mathbf{u}) \implies C\mathbf{u} = \lambda\mathbf{u}$$

3. Calculus Proof: Differentiating the Lagrangian L(u, λ)

To write this algebraically, we form the Lagrangian function $\mathcal{L}(\mathbf{u}, \lambda)$ by subtracting $\lambda(\mathbf{u}^T\mathbf{u} - 1)$ from our objective:

$$\mathcal{L}(\mathbf{u}, \lambda) = \mathbf{u}^T C \mathbf{u} - \lambda(\mathbf{u}^T \mathbf{u} - 1)$$

Using the matrix calculus identities $\nabla_{\mathbf{u}}(\mathbf{u}^T C \mathbf{u}) = 2C\mathbf{u}$ (for symmetric $C$) and $\nabla_{\mathbf{u}}(\mathbf{u}^T \mathbf{u}) = 2\mathbf{u}$, we set the gradient with respect to $\mathbf{u}$ to zero:

4. The Payoff: Proving uᵀCu = λ and Ranking Principal Components

A symmetric $d \times d$ matrix $C$ has $d$ orthonormal eigenvectors $\mathbf{u}_1, \dots, \mathbf{u}_d$ with eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d \ge 0$. Which eigenvector maximizes $\mathbf{u}^T C \mathbf{u}$? Left-multiply $C\mathbf{u} = \lambda\mathbf{u}$ by $\mathbf{u}^T$ and use $\mathbf{u}^T\mathbf{u} = 1$:

$$\sigma_{\mathbf{u}}^2 = \mathbf{u}^T C \mathbf{u} = \mathbf{u}^T (\lambda \mathbf{u}) = \lambda (\mathbf{u}^T \mathbf{u}) = \lambda(1) = \lambda$$

The eigenvalue $\lambda$ IS the projected variance! Therefore:

• The eigenvector $\mathbf{u}_1$ with the largest eigenvalue $\lambda_1$ is the global maximum of variance (and global minimum of orthogonal reconstruction error)—this is Principal Component 1 (PC1).
• The eigenvector $\mathbf{u}_d$ with the smallest eigenvalue $\lambda_d$ is the direction of minimum variance.

5. Deflation: Why the 2nd Principal Component Is the 2nd Eigenvector

To find the second principal component $\mathbf{u}_2$, we maximize $\mathbf{u}^T C \mathbf{u}$ subject to both $\mathbf{u}^T\mathbf{u} = 1$ and orthogonality to the first component, $\mathbf{u}^T \mathbf{u}_1 = 0$. Forming the two-constraint Lagrangian $\mathcal{L} = \mathbf{u}^T C \mathbf{u} - \lambda(\mathbf{u}^T\mathbf{u} - 1) - \mu(\mathbf{u}^T \mathbf{u}_1)$ and differentiating yields $2C\mathbf{u} - 2\lambda\mathbf{u} - \mu\mathbf{u}_1 = \mathbf{0}$. Taking the dot product with $\mathbf{u}_1$ shows $\mu = 0$, leaving $C\mathbf{u} = \lambda\mathbf{u}$ among directions orthogonal to $\mathbf{u}_1$—whose best solution is $(\lambda_2, \mathbf{u}_2)$!

6. Complete Worked Numerical Example

Suppose a centered 2D dataset has total scatter $\frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i\|^2 = 8$ and covariance matrix $C = \begin{bmatrix} 5 & 2 \\ 2 & 3 \end{bmatrix}$.

1. Solve $\det(C - \lambda I) = 0$: $\lambda^2 - 8\lambda + (15 - 4) = \lambda^2 - 8\lambda + 11 = 0$, giving $\lambda_1 = 4 + \sqrt{5} \approx 6.236$ and $\lambda_2 = 4 - \sqrt{5} \approx 1.764$.
2. Maximum Variance & Minimum Error along PC1 ($\mathbf{u}_1$):
   • Projected variance along $\mathbf{u}_1$: $\sigma_{\mathbf{u}_1}^2 = \lambda_1 \approx 6.236$.
   • Mean squared reconstruction error when compressing to PC1 (by Chapter 2 Pythagorean theorem): $J(\mathbf{u}_1) = \text{tr}(C) - \lambda_1 = 8 - 6.236 = \lambda_2 \approx 1.764$!

Notice that remarkable duality: when you keep the top $k$ principal components, the captured variance is $\sum_{j=1}^k \lambda_j$ and the discarded reconstruction error is the sum of the discarded eigenvalues $\sum_{j=k+1}^d \lambda_j$!

7. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: A 4D dataset has covariance eigenvalues $\lambda_1 = 12, \lambda_2 = 5, \lambda_3 = 2, \lambda_4 = 1$. If we project the dataset onto the 2D plane spanned by PC1 and PC2, what is the captured variance and what is the mean squared orthogonal reconstruction error?

Answer: The captured variance is $\lambda_1 + \lambda_2 = 12 + 5 = 17$, and by the Pythagorean theorem the mean squared orthogonal reconstruction error is the sum of the discarded eigenvalues: $\lambda_3 + \lambda_4 = 2 + 1 = 3$.

• Chapter Microlearnings: Dive deeper into 5.1 Why uᵀu = 1 Is Essential, 5.2 Kissing Contours and Lagrange Multipliers, and 5.3 Why λ Equals Projected Variance.
• Canonical References: See Wikipedia Reference on the Rayleigh Quotient and Wikipedia Reference on Lagrange Multipliers.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 5 Overview: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation
• Microlearning 5.1: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential
• Microlearning 5.2: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically
• Microlearning 5.3: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u
• Previous Lesson: 4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors
• Next Lesson: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential