5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically

How does a constrained optimization problem turn into an eigenvalue problem? The bridge is Lagrange's method of multipliers, which rests on a vivid geometric principle: at an extremum on a constraint surface, the level contour of the objective function must kiss the constraint surface tangentially, forcing their gradients to line up.

In this microlearning, we derive the matrix gradients $\nabla_{\mathbf{u}}(\mathbf{u}^T C \mathbf{u}) = 2C\mathbf{u}$ and $\nabla_{\mathbf{u}}(\mathbf{u}^T \mathbf{u}) = 2\mathbf{u}$ from first principles and watch their alignment produce $C\mathbf{u} = \lambda\mathbf{u}$.

In this Article

• 1. Why Kissing Contours Have Parallel Gradients
• 2. Deriving ∇(uᵀCu) = 2Cu and ∇(uᵀu) = 2u Coordinate by Coordinate
• 3. Setting Up the Lagrangian andCanceling the Factor of 2
• 4. Worked Gradient Alignment Check in 2D
• 5. Active Recall Check and Key Takeaways

1. Why Kissing Contours Have Parallel Gradients

Recall from multivariable calculus that the gradient $\nabla f(\mathbf{u})$ is always perpendicular to the level curve $f(\mathbf{u}) = \text{const}$, and points in the direction of steepest ascent. Suppose you are restricted to walk on the unit circle $g(\mathbf{u}) = \mathbf{u}^T \mathbf{u} = 1$:

• If $\nabla f(\mathbf{u})$ is not parallel to the circle normal $\nabla g(\mathbf{u})$, then $\nabla f(\mathbf{u})$ has a non-zero component tangent to the circle. Moving slightly along that tangent increases $f(\mathbf{u})$, so you are not at a maximum!
• Therefore, at any constrained maximum or minimum on the unit circle, $\nabla f(\mathbf{u})$ can have zero tangential component—it must be purely normal to the circle, meaning $\nabla f(\mathbf{u}) = \lambda \nabla g(\mathbf{u})$ for some scalar multiplier $\lambda$.

   NOT OPTIMAL (Crossing Contours):               OPTIMAL (Kissing Contours at Eigenvector):
         Grad f  ^                                     Grad f = 2 C u  ^
                /                                                      |  Grad g = 2 u
               /---> Tangent component != 0                            |  ^
   -----------*----------- (Circle Tangent)       ---------------------*--|----------------
              |                                                        (Zero tangent component!)
              v Grad g                                                 2 C u = lambda (2 u)

2. Deriving ∇(uᵀCu) = 2Cu and ∇(uᵀu) = 2u Coordinate by Coordinate

You do not need to memorize mysterious matrix calculus tables—let us verify both derivatives directly from partial derivatives:

3. Setting Up the Lagrangian and Canceling the Factor of 2

Combining the objective and constraint into the Lagrangian $\mathcal{L}(\mathbf{u}, \lambda) = \mathbf{u}^T C \mathbf{u} - \lambda(\mathbf{u}^T \mathbf{u} - 1)$ and setting $\nabla_{\mathbf{u}} \mathcal{L} = \mathbf{0}$ immediately yields:

$$2C\mathbf{u} - 2\lambda\mathbf{u} = \mathbf{0} \implies C\mathbf{u} = \lambda\mathbf{u}$$

The factor of $2$ from both quadratic derivatives cancels cleanly, leaving the exact eigenvalue equation!

4. Worked Gradient Alignment Check in 2D

Let $C = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$. Let us compare the gradients $\nabla f = 2C\mathbf{u}$ and $\nabla g = 2\mathbf{u}$ at two points on the unit circle:

1. At non-eigenvector $\mathbf{u} = (1, 0)^T$: $\nabla g = (2, 0)^T$, while $\nabla f = 2C\mathbf{u} = (6, 2)^T$. Because $(6, 2)^T$ is not a scalar multiple of $(2, 0)^T$, the gradients are misaligned and increasing $u_2$ increases variance.
2. At eigenvector $\mathbf{u}_1 = (1/\sqrt{2}, 1/\sqrt{2})^T$: $\nabla g = \sqrt{2}(1, 1)^T$, and $\nabla f = 2C\mathbf{u}_1 = 4\sqrt{2}(1, 1)^T = 4 \nabla g$. The gradients are strictly parallel with multiplier $\lambda_1 = 4$!

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: If we wrote the Rayleigh quotient $R(\mathbf{w}) = \frac{\mathbf{w}^T C \mathbf{w}}{\mathbf{w}^T \mathbf{w}}$ and differentiated with respect to $\mathbf{w}$ using the quotient rule, what equation would $\nabla_{\mathbf{w}} R(\mathbf{w}) = \mathbf{0}$ produce?

Answer: Setting $\nabla_{\mathbf{w}} R(\mathbf{w}) = \frac{2C\mathbf{w}(\mathbf{w}^T\mathbf{w}) - (\mathbf{w}^T C \mathbf{w})(2\mathbf{w})}{(\mathbf{w}^T\mathbf{w})^2} = \mathbf{0}$ gives $C\mathbf{w} = R(\mathbf{w})\mathbf{w}$, which is the exact same eigenvalue equation $C\mathbf{w} = \lambda\mathbf{w}$ with $\lambda = R(\mathbf{w})$!

• Geometric Meaning of Lagrange Multiplier: Kissing level sets require $\nabla f = \lambda \nabla g$.
• Quadratic Derivative Rule: For symmetric $C$, $\nabla_{\mathbf{u}}(\mathbf{u}^T C \mathbf{u}) = 2C\mathbf{u}$.
• Canonical References: See Wikipedia Reference on Matrix Calculus Identities.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 5 Overview: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation
• Microlearning 5.1: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential
• Microlearning 5.2: 5.2 Kissing Contours and Lagrange Multipliers: Deriving Cu = λu Geometrically
• Microlearning 5.3: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u
• Previous Lesson: 5.1 Why Unconstrained Variance Maximization Fails and Why uᵀu = 1 Is Essential
• Next Lesson: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u