5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential

One of the most frequent questions students ask when seeing the PCA optimization problem for the first time is: "Why do we need the constraint $\mathbf{u}^T \mathbf{u} = 1$? Why can't we just set the derivative of $\mathbf{u}^T C \mathbf{u}$ to zero directly?"

In this microlearning, we see exactly what goes wrong when you remove the unit-norm constraint—both algebraically and geometrically—and introduce the equivalent scale-free formulation known as the Rayleigh quotient.

In this Article

• 1. The Quadratic Scaling Trap: Scaling u by α Scales uᵀCu by α²
• 2. Why Setting ∇(uᵀCu) = 0 Directly Gives Only u = 0
• 3. The Rayleigh Quotient R(w) = (wᵀCw)/(wᵀw): Built-In Normalization
• 4. Worked Numerical Comparison
• 5. Active Recall Check and Key Takeaways

1. The Quadratic Scaling Trap: Scaling u by α Scales uᵀCu by α²

Suppose you fix a direction in space via a unit vector $\mathbf{u}_0$ with true directional variance $\mathbf{u}_0^T C \mathbf{u}_0 = 3$. Now consider what happens if you simply stretch the ruler by defining $\mathbf{w} = \alpha \mathbf{u}_0$ for any scalar $\alpha > 0$:

$$\mathbf{w}^T C \mathbf{w} = (\alpha \mathbf{u}_0)^T C (\alpha \mathbf{u}_0) = \alpha^2 (\mathbf{u}_0^T C \mathbf{u}_0) = 3\alpha^2$$

   Length ||w|| = 1          Length ||w|| = 10           Length ||w|| -> infinity
   w^T C w = 3               w^T C w = 300               w^T C w -> +infinity!
   (True Variance)           (Fake 100x Inflation!)      (Unbounded Explosion!)

Without restricting $\|\mathbf{w}\| = 1$, an optimizer can drive $\mathbf{w}^T C \mathbf{w}$ to $+\infty$ along any non-zero variance direction just by making $\|\mathbf{w}\|$ larger and larger. That tells us nothing about which direction in the data cloud has the greatest spread!

2. Why Setting ∇(uᵀCu) = 0 Directly Gives Only u = 0

What if you try to find a stationary point of $f(\mathbf{u}) = \mathbf{u}^T C \mathbf{u}$ without constraints by setting $\nabla f(\mathbf{u}) = 2C\mathbf{u} = \mathbf{0}$? When $C$ is positive definite, the only solution to $2C\mathbf{u} = \mathbf{0}$ is $\mathbf{u} = \mathbf{0}$—the bottom of the quadratic bowl where variance is zero (the global minimum), not the maximum!

3. The Rayleigh Quotient R(w) = (wᵀCw)/(wᵀw): Built-In Normalization

There are two mathematically identical ways to prevent length inflation:

4. Worked Numerical Comparison

Let $C = \begin{bmatrix} 4 & 0 \\ 0 & 1 \end{bmatrix}$. Compare a short vector along the high-variance $x_1$ axis, $\mathbf{w}_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$, against a stretched vector along the low-variance $x_2$ axis, $\mathbf{w}_2 = \begin{bmatrix} 0 \\ 5 \end{bmatrix}$:

• Unnormalized quadratic form: $\mathbf{w}_1^T C \mathbf{w}_1 = 4$ vs. $\mathbf{w}_2^T C \mathbf{w}_2 = 25$. The unnormalized form is fooled into picking $\mathbf{w}_2$ solely because $\|\mathbf{w}_2\| = 5$!
• Normalized Rayleigh quotient (or unit sphere $\mathbf{u}^T\mathbf{u}=1$): $R(\mathbf{w}_1) = 4/1 = 4$ vs. $R(\mathbf{w}_2) = 25/25 = 1$. Normalization exposes the true directional variances $4$ vs. $1$.

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: If $\mathbf{w} = (3, 4)^T$ and $\mathbf{w}^T C \mathbf{w} = 100$, what is the true projected variance of the data along the direction of $\mathbf{w}$?

Answer: Since $\mathbf{w}^T \mathbf{w} = 3^2 + 4^2 = 25$, the unit vector is $\mathbf{u} = \mathbf{w}/5$ and the true directional variance is the Rayleigh quotient $R(\mathbf{w}) = \frac{\mathbf{w}^T C \mathbf{w}}{\mathbf{w}^T \mathbf{w}} = \frac{100}{25} = 4$.

• Why $\mathbf{u}^T\mathbf{u}=1$ Exists: It separates pure geometric direction from artificial coordinate scaling.
• Rayleigh Quotient Equivalence: Maximizing $\mathbf{u}^T C \mathbf{u}$ on $\mathbf{u}^T\mathbf{u}=1$ is identical to maximizing $(\mathbf{w}^T C \mathbf{w})/(\mathbf{w}^T\mathbf{w})$ over $\mathbf{w} \neq \mathbf{0}$.
• Canonical References: See Wikipedia Reference on the Rayleigh Quotient.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 5 Overview: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation
• Microlearning 5.1: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential
• Microlearning 5.2: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically
• Microlearning 5.3: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u
• Previous Lesson: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation
• Next Lesson: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically