5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u

When solving $\nabla_{\mathbf{u}} [\mathbf{u}^T C \mathbf{u} - \lambda(\mathbf{u}^T\mathbf{u} - 1)] = \mathbf{0}$, the condition $C\mathbf{u} = \lambda\mathbf{u}$ is satisfied by every eigenvector of $C$—including the eigenvector of minimum variance, the eigenvector of maximum variance, and all saddle-point eigenvectors in between.

How do we know which eigenvector is the global maximum of variance, and what physical meaning does the Lagrange multiplier $\lambda$ carry? One line of algebra reveals that $\lambda$ is not an abstract book-keeping constant: $\lambda$ equals the exact projected variance of the dataset along $\mathbf{u}$.

In this Article

• 1. The One-Line Proof That uᵀCu = λ for Any Unit Eigenvector
• 2. Maximum, Minimum, and Saddle Points on the Unit Sphere
• 3. Proof That No Other Unit Vector Can Beat λ₁
• 4. Worked 3D Eigenvalue Variance & Error Table
• 5. Active Recall Check and Key Takeaways

1. The One-Line Proof That uᵀCu = λ for Any Unit Eigenvector

Take any unit eigenvector $\mathbf{u}_k$ of $C$ satisfying both $C\mathbf{u}_k = \lambda_k \mathbf{u}_k$ and $\mathbf{u}_k^T \mathbf{u}_k = 1$. Substitute $C\mathbf{u}_k = \lambda_k \mathbf{u}_k$ directly into the directional variance formula $\sigma_{\mathbf{u}_k}^2 = \mathbf{u}_k^T C \mathbf{u}_k$:

2. Maximum, Minimum, and Saddle Points on the Unit Sphere

In $d$ dimensions, order the eigenvalues from largest to smallest: $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d \ge 0$. On the unit sphere $S^{d-1}$, every unit eigenvector $\pm \mathbf{u}_k$ is a stationary point of $f(\mathbf{u}) = \mathbf{u}^T C \mathbf{u}$:

   On a 3D Unit Sphere (lambda_1 > lambda_2 > lambda_3):
   - u_1 (largest eigenvalue lambda_1)  ---> Global MAXIMUM of variance (Mountain Peak on sphere)
   - u_2 (middle eigenvalue lambda_2)   ---> SADDLE POINT on sphere (Max in u_2-u_3 plane, Min in u_1-u_2 plane)
   - u_3 (smallest eigenvalue lambda_3) ---> Global MINIMUM of variance (Valley Bottom on sphere)

3. Proof That No Other Unit Vector Can Beat λ₁

How can we be 100% certain that no non-eigenvector unit direction $\mathbf{u}$ could sneak past $\lambda_1$? Expand any unit vector $\mathbf{u} \in \mathbb{R}^d$ in the orthonormal eigenbasis $\mathbf{u} = \sum_{j=1}^d c_j \mathbf{u}_j$, where $\sum_{j=1}^d c_j^2 = 1$. Then:

$$\mathbf{u}^T C \mathbf{u} = \sum_{j=1}^d \lambda_j c_j^2 \le \lambda_1 \sum_{j=1}^d c_j^2 = \lambda_1(1) = \lambda_1$$

Since $\mathbf{u}^T C \mathbf{u}$ is a convex weighted average of the eigenvalues $\{\lambda_1, \dots, \lambda_d\}$ with weights $c_j^2 \ge 0$ summing to $1$, it is bounded between $\lambda_d \le \mathbf{u}^T C \mathbf{u} \le \lambda_1$, achieving $\lambda_1$ when $c_1 = \pm 1$ (i.e., $\mathbf{u} = \pm \mathbf{u}_1$)!

4. Worked 3D Eigenvalue Variance & Error Table

Suppose a 3D dataset (measured in $\text{cm}$) has covariance eigenvalues $\lambda_1 = 9.0\text{ cm}^2$, $\lambda_2 = 4.0\text{ cm}^2$, and $\lambda_3 = 1.0\text{ cm}^2$ (total variance $= 14.0\text{ cm}^2$).

Subspace Kept	Captured Variance ($\text{cm}^2$)	Standard Deviation Along New Axis	Mean Squared Orthogonal Error ($\text{cm}^2$)
1D Line ($\mathbf{u}_1$ only)	$$\lambda_1 = 9.0$$	$$\sqrt{\lambda_1} = 3.0\text{ cm}$$	$$\lambda_2 + \lambda_3 = 4.0 + 1.0 = 5.0$$
2D Plane ($\mathbf{u}_1, \mathbf{u}_2$)	$$\lambda_1 + \lambda_2 = 13.0$$	$$\sqrt{\lambda_2} = 2.0\text{ cm}$$	$$\lambda_3 = 1.0$$
Full 3D Space ($\mathbf{u}_1, \mathbf{u}_2, \mathbf{u}_3$)	$$\lambda_1 + \lambda_2 + \lambda_3 = 14.0$$	$$\sqrt{\lambda_3} = 1.0\text{ cm}$$	$$0.0$$

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: If a unit vector $\mathbf{u}$ is an equal $50\%/50\%$ superposition of $\mathbf{u}_1$ and $\mathbf{u}_2$, namely $\mathbf{u} = \frac{1}{\sqrt{2}}\mathbf{u}_1 + \frac{1}{\sqrt{2}}\mathbf{u}_2$, what is the projected variance $\mathbf{u}^T C \mathbf{u}$?

Answer: Since $c_1^2 = 1/2$ and $c_2^2 = 1/2$, the projected variance is the exact average of the two eigenvalues: $\mathbf{u}^T C \mathbf{u} = \frac{1}{2}\lambda_1 + \frac{1}{2}\lambda_2$.

• Physical Meaning of $\lambda_k$: $\lambda_k = \mathbf{u}_k^T C \mathbf{u}_k$ is the variance along principal component $k$, and $\sqrt{\lambda_k}$ is the standard deviation along that axis.
• Convex Combination Bound: Every unit vector's variance is a weighted average $\sum c_j^2 \lambda_j$ bounded between $\lambda_{\min}$ and $\lambda_{\max}$.
• Canonical References: See Wikipedia Reference on the Courant–Fischer Min-Max Theorem.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 5 Overview: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation
• Microlearning 5.1: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential
• Microlearning 5.2: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically
• Microlearning 5.3: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u
• Previous Lesson: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically
• Next Lesson: Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD