4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction

Once we have built the covariance matrix $C = \frac{1}{n}X^T X$, how do we find out how spread out the data cloud is along an arbitrary tilted direction $\mathbf{u}$? The answer is one of the most important expressions in applied mathematics: the quadratic form $\mathbf{u}^T C \mathbf{u}$.

In this microlearning, we prove that $\mathbf{u}^T C \mathbf{u}$ equals the exact empirical variance along unit vector $\mathbf{u}$, expand it into a polynomial in the components of $\mathbf{u}$, and visualize its level sets as variance ellipses.

In this Article

• 1. Proving σ²(u) = uᵀCu in Three Lines of Matrix Algebra
• 2. Expanding uᵀCu in 2D: How Cross-Terms Reward Alignment
• 3. Visualizing the Quadratic Bowl and Variance Contours
• 4. Worked Numerical Directional Sweep
• 5. Active Recall Check and Key Takeaways

1. Proving σ²(u) = uᵀCu in Three Lines of Matrix Algebra

Let $X \in \mathbb{R}^{n \times d}$ be our centered data matrix and $\mathbf{u} \in \mathbb{R}^d$ be any unit direction vector ($\mathbf{u}^T\mathbf{u} = 1$). Projecting each row $\mathbf{x}_i^T$ onto $\mathbf{u}$ produces the vector of $n$ scalar projections:

$$\mathbf{z} = X\mathbf{u} = \begin{bmatrix} \mathbf{x}_1^T \mathbf{u} \\ \vdots \\ \mathbf{x}_n^T \mathbf{u} \end{bmatrix}$$

Because $X$ is mean-centered, the average of $\mathbf{z}$ is zero. Its empirical variance $\sigma_{\mathbf{u}}^2$ is the average squared entry of $\mathbf{z}$:

$$\sigma_{\mathbf{u}}^2 = \frac{1}{n} \mathbf{z}^T \mathbf{z} = \frac{1}{n} (X\mathbf{u})^T (X\mathbf{u}) = \mathbf{u}^T \left(\frac{1}{n} X^T X\right) \mathbf{u} = \mathbf{u}^T C \mathbf{u}$$

2. Expanding uᵀCu in 2D: How Cross-Terms Reward Alignment

In 2D, let $\mathbf{u} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} \cos\theta \\ \sin\theta \end{bmatrix}$ and $C = \begin{bmatrix} c_{11} & c_{12} \\ c_{12} & c_{22} \end{bmatrix}$. Carrying out the multiplication $\mathbf{u}^T C \mathbf{u}$ gives a degree-2 homogeneous polynomial (which is why it is called a quadratic form):

3. Visualizing the Quadratic Bowl and Variance Contours

If we allow an arbitrary vector $\mathbf{w} \in \mathbb{R}^2$ and plot $f(\mathbf{w}) = \mathbf{w}^T C \mathbf{w}$ as a height in 3D, it forms an upward-curving elliptic paraboloid (a bowl). Slicing this bowl horizontally at a constant height $f(\mathbf{w}) = k > 0$ produces concentric ellipses in the plane:

            w2 ^
               |          . - ~ ~ - .
               |      . '            / ' .   <-- Level curve w^T C w = constant
               |    /               /     \
         ------+---/---------------0-------+---> w1
               |  \               /       /
               |    ' .          /    . '
               |        ' - . _ . - '
                 Minor axis = Steepest wall of bowl (largest eigenvalue lambda_1)
                 Major axis = Gentlest valley of bowl (smallest eigenvalue lambda_2)

4. Worked Numerical Directional Sweep

Suppose $C = \begin{bmatrix} 4 & 2 \\ 2 & 4 \end{bmatrix}$. Then for any unit vector $\mathbf{u} = (\cos\theta, \sin\theta)^T$:

$$\mathbf{u}^T C \mathbf{u} = 4\cos^2\theta + 4\cos\theta\sin\theta + 4\sin^2\theta = 4 + 2\sin(2\theta)$$

Angle $\theta$	Unit Vector $\mathbf{u}$	Directional Variance $\mathbf{u}^T C \mathbf{u} = 4 + 2\sin(2\theta)$
$$0^\circ$$	$$(1, 0)^T$$	$$4 + 2(0) = 4.0$$
$$45^\circ$$	$$(1/\sqrt{2}, 1/\sqrt{2})^T$$	$4 + 2(1) = 6.0$ (Global Maximum = $\lambda_1$)
$$90^\circ$$	$$(0, 1)^T$$	$$4 + 2(0) = 4.0$$
$$135^\circ$$	$$(-1/\sqrt{2}, 1/\sqrt{2})^T$$	$4 + 2(-1) = 2.0$ (Global Minimum = $\lambda_2$)

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: Given $C = \begin{bmatrix} 5 & -3 \\ -3 & 5 \end{bmatrix}$, does a unit vector in Quadrant I ($u_1 > 0, u_2 > 0$) or Quadrant II ($u_1 < 0, u_2 > 0$) achieve larger projected variance?

Answer: Quadrant II! Because $c_{12} = -3 < 0$, the cross term $-6 u_1 u_2$ becomes positive (+6 u_1 |u_2|) when $u_1$ and $u_2$ have opposite signs, yielding maximum variance $8$ at $\mathbf{u} = (-1/\sqrt{2}, 1/\sqrt{2})^T$.

• Universal Variance Formula: $\sigma_{\mathbf{u}}^2 = \mathbf{u}^T C \mathbf{u}$ for any unit vector $\mathbf{u}$.
• Cross-Term Geometry: Off-diagonal covariances reward directions aligned with the correlation sign of the cloud.
• Canonical References: See Wikipedia Reference on Quadratic Forms in Statistics.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 4 Overview: Chapter 4: What a Covariance Matrix Really Is — From Scatter Clouds to Orthogonal Stretch Axes
• Microlearning 4.1: 4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data
• Microlearning 4.2: 4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction
• Microlearning 4.3: 4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors
• Previous Lesson: 4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data
• Next Lesson: 4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors