4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors

Why are the principal components of a dataset guaranteed to be mutually perpendicular (orthogonal)? After all, we saw in Chapter 3 that a general asymmetric matrix like $\begin{bmatrix} 1 & 1 \\ 0 & 2 \end{bmatrix}$ has non-orthogonal eigenvectors. What makes a covariance matrix special?

The answer is symmetry: $C^T = C$. The Real Spectral Theorem proves that every real symmetric matrix has exclusively real eigenvalues and admits an orthonormal basis of eigenvectors. Better yet, the proof of orthogonality takes just four lines of elementary dot-product algebra.

In this Article

• 1. The Self-Adjoint Property: Moving C Across a Dot Product
• 2. The Four-Line Proof That Distinct Eigenvectors Are Orthogonal
• 3. Spectral Decomposition C = QΛQᵀ: Rotate, Stretch, Rotate Back
• 4. Worked Spectral Factorization Example
• 5. Active Recall Check and Key Takeaways

1. The Self-Adjoint Property: Moving C Across a Dot Product

For any two vectors $\mathbf{u}, \mathbf{v} \in \mathbb{R}^d$ and any matrix $A$, the dot product between $A\mathbf{u}$ and $\mathbf{v}$ satisfies $(A\mathbf{u}) \cdot \mathbf{v} = (A\mathbf{u})^T \mathbf{v} = \mathbf{u}^T A^T \mathbf{v}$. When $C$ is symmetric ($C^T = C$), $C$ can move freely from one side of the dot product to the other:

$$(C\mathbf{u})^T \mathbf{v} = \mathbf{u}^T C^T \mathbf{v} = \mathbf{u}^T (C\mathbf{v})$$

2. The Four-Line Proof That Distinct Eigenvectors Are Orthogonal

Suppose $\mathbf{u}_1$ and $\mathbf{u}_2$ are eigenvectors of a symmetric matrix $C$ with distinct eigenvalues $\lambda_1 \neq \lambda_2$, so $C\mathbf{u}_1 = \lambda_1 \mathbf{u}_1$ and $C\mathbf{u}_2 = \lambda_2 \mathbf{u}_2$. Let us compute $\mathbf{u}_1^T C \mathbf{u}_2$ in two ways:

3. Spectral Decomposition C = QΛQᵀ: Rotate, Stretch, Rotate Back

Stacking the orthonormal eigenvectors $\mathbf{u}_1, \dots, \mathbf{u}_d$ as columns of an orthogonal matrix $Q$ (so $Q^T Q = Q Q^T = I$) and placing $\lambda_1, \dots, \lambda_d$ on the diagonal of $\Lambda$ yields the Spectral Decomposition:

$$C = Q \Lambda Q^T = \sum_{j=1}^d \lambda_j \mathbf{u}_j \mathbf{u}_j^T$$

   Action of C = Q * Lambda * Q^T on any vector v:
   [ Input v ] ---( Q^T )---> [ Rotate into Eigen-Axes ] ---( Lambda )---> [ Pure Axis Stretch ] ---( Q )---> [ Rotate Back ]

4. Worked Spectral Factorization Example

Take $C = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix}$. From Chapter 3, its eigenvalues are $\lambda_1 = 6$ and $\lambda_2 = 1$ with eigenvectors $\mathbf{v}_1 = (2, 1)^T$ and $\mathbf{v}_2 = (-1, 2)^T$.

1. Verify Orthogonality: $\mathbf{v}_1^T \mathbf{v}_2 = (2)(-1) + (1)(2) = 0$.
2. Normalize to Unit Vectors: $\mathbf{u}_1 = \frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}$ and $\mathbf{u}_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}$.
3. Verify Rank-1 Sum $C = \lambda_1 \mathbf{u}_1 \mathbf{u}_1^T + \lambda_2 \mathbf{u}_2 \mathbf{u}_2^T$:

   $$6 \cdot \frac{1}{5}\begin{bmatrix} 4 & 2 \\ 2 & 1 \end{bmatrix} + 1 \cdot \frac{1}{5}\begin{bmatrix} 1 & -2 \\ -2 & 4 \end{bmatrix} = \frac{1}{5}\begin{bmatrix} 25 & 10 \\ 10 & 10 \end{bmatrix} = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix} = C$$

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: What happens if a symmetric covariance matrix has a repeated eigenvalue, say $\lambda_1 = \lambda_2 = 4$ (which happens when a 2D cloud is perfectly circular)?

Answer: Every nonzero vector in that 2D eigenspace is an eigenvector with variance $4$, so we are free to pick any two perpendicular unit vectors (e.g., via Gram-Schmidt or standard axes) to form an orthonormal basis.

• Symmetry Guarantees Orthogonality: $C^T = C$ forces eigenvectors of distinct eigenvalues to be perpendicular.
• Decorrelation: Rotating data into the eigenbasis $Z = XQ$ diagonalizes the covariance matrix to $\Lambda$, making the principal component scores completely uncorrelated.
• Canonical References: See Wikipedia Reference on the Real Spectral Theorem.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 4 Overview: Chapter 4: What a Covariance Matrix Really Is — From Scatter Clouds to Orthogonal Stretch Axes
• Microlearning 4.1: 4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data
• Microlearning 4.2: 4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction
• Microlearning 4.3: 4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors
• Previous Lesson: 4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction
• Next Lesson: Chapter 5: The Grand Unification — How Variance Maximization Births the Eigenvalue Equation