2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error

In visual diagrams of Principal Component Analysis, we draw a right-angled square where the residual line meets the candidate axis $\mathbf{u}$. But how do we prove algebraically in $d$-dimensional space ($\mathbb{R}^d$, where we cannot draw pictures) that the residual vector $\mathbf{r}_i = \mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$ is strictly perpendicular to the projection vector $\hat{\mathbf{x}}_i = (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$, and that the cross-term vanishes in the squared norm expansion?

In this Article

• 1. Algebraic Proof: The Residual Vector Is Orthogonal to the Axis
• 2. Expanding the Squared Error Norm via Dot Products
• 3. Generalizing to a $k$-Dimensional Principal Subspace
• 4. Worked 3D Numerical Verification
• 5. Active Recall Check and Key Takeaways

1. Algebraic Proof: The Residual Vector Is Orthogonal to the Axis

Let $\mathbf{x}_i \in \mathbb{R}^d$ be any vector and $\mathbf{u} \in \mathbb{R}^d$ be any unit vector ($\mathbf{u}^T\mathbf{u} = 1$). Define the scalar projection $z_i = \mathbf{u}^T\mathbf{x}_i$, the shadow vector $\hat{\mathbf{x}}_i = z_i \mathbf{u}$, and the residual error vector $\mathbf{r}_i = \mathbf{x}_i - z_i \mathbf{u}$.

Two vectors are orthogonal if and only if their dot product is zero. Let us take the dot product of $\mathbf{u}$ with $\mathbf{r}_i$:

$$\mathbf{u}^T \mathbf{r}_i = \mathbf{u}^T (\mathbf{x}_i - z_i \mathbf{u}) = \mathbf{u}^T \mathbf{x}_i - z_i (\mathbf{u}^T \mathbf{u}) = z_i - z_i(1) = 0$$

Because $\mathbf{u}^T \mathbf{r}_i = 0$, the residual $\mathbf{r}_i$ is also orthogonal to the shadow vector $\hat{\mathbf{x}}_i = z_i \mathbf{u}$: $\hat{\mathbf{x}}_i^T \mathbf{r}_i = z_i (\mathbf{u}^T \mathbf{r}_i) = 0$. This confirms a $90^\circ$ angle in any dimension $d$!

2. Expanding the Squared Error Norm via Dot Products

Even if you did not notice the right triangle geometrically, you can derive the Pythagorean decomposition directly by expanding the squared reconstruction error $e_i^2 = \|\mathbf{x}_i - z_i\mathbf{u}\|^2$:

$$\begin{aligned} e_i^2 &= (\mathbf{x}_i - z_i\mathbf{u})^T (\mathbf{x}_i - z_i\mathbf{u}) \ &= \mathbf{x}_i^T \mathbf{x}_i - 2 z_i (\mathbf{u}^T \mathbf{x}_i) + z_i^2 (\mathbf{u}^T \mathbf{u}) \end{aligned}$$

Now substitute $\mathbf{u}^T \mathbf{x}_i = z_i$ in the middle term and $\mathbf{u}^T \mathbf{u} = 1$ in the last term:

$$e_i^2 = \|\mathbf{x}_i\|^2 - 2 z_i^2 + z_i^2(1) = \|\mathbf{x}_i\|^2 - z_i^2 = \|\mathbf{x}_i\|^2 - (\mathbf{u}^T\mathbf{x}_i)^2$$

3. Generalizing to a $k$-Dimensional Principal Subspace

What if we project onto a $k$-dimensional plane spanned by $k$ mutually orthonormal vectors $\mathbf{u}_1, \mathbf{u}_2, \dots, \mathbf{u}_k$ (where $\mathbf{u}_a^T \mathbf{u}_b = 0$ for $a \neq b$ and $\mathbf{u}_a^T \mathbf{u}_a = 1$)? The shadow is $\hat{\mathbf{x}}_i = \sum_{a=1}^k (\mathbf{u}_a^T\mathbf{x}_i)\mathbf{u}_a$, and the exact same expansion gives:

$$e_i^2 = \|\mathbf{x}_i\|^2 - \sum_{a=1}^k (\mathbf{u}_a^T\mathbf{x}_i)^2$$

Each additional orthogonal axis $\mathbf{u}_a$ simply subtracts its own independent squared projection $( \mathbf{u}_a^T\mathbf{x}_i )^2$ from the remaining error!

4. Worked 3D Numerical Verification

Let $\mathbf{x}_i = \begin{bmatrix} 2 \\ 3 \\ 6 \end{bmatrix} \in \mathbb{R}^3$ (so $\|\mathbf{x}_i\|^2 = 4 + 9 + 36 = 49$), and let $\mathbf{u} = \begin{bmatrix} 2/3 \\ 1/3 \\ 2/3 \end{bmatrix}$ (unit vector since $\frac{4+1+4}{9} = 1$).

1. Scalar Projection: $z_i = \mathbf{u}^T\mathbf{x}_i = \frac{2(2) + 1(3) + 2(6)}{3} = \frac{4+3+12}{3} = \frac{19}{3}$, so $z_i^2 = \frac{361}{9}$.
2. Shadow & Residual Vectors: $\hat{\mathbf{x}}_i = \frac{19}{3}\begin{bmatrix} 2/3 \\ 1/3 \\ 2/3 \end{bmatrix} = \frac{1}{9}\begin{bmatrix} 38 \\ 19 \\ 38 \end{bmatrix}$, and $\mathbf{r}_i = \mathbf{x}_i - \hat{\mathbf{x}}_i = \frac{1}{9}\begin{bmatrix} 18-38 \\ 27-19 \\ 54-38 \end{bmatrix} = \frac{1}{9}\begin{bmatrix} -20 \\ 8 \\ 16 \end{bmatrix}$.
3. Verify Orthogonality: $\mathbf{u}^T \mathbf{r}_i = \frac{1}{27}(2(-20) + 1(8) + 2(16)) = \frac{-40 + 8 + 32}{27} = 0$.
4. Verify Pythagoras: $e_i^2 = \|\mathbf{r}_i\|^2 = \frac{(-20)^2 + 8^2 + 16^2}{81} = \frac{400 + 64 + 256}{81} = \frac{720}{81} = \frac{80}{9}$. Adding base and error squared: $z_i^2 + e_i^2 = \frac{361}{9} + \frac{80}{9} = \frac{441}{9} = 49 = \|\mathbf{x}_i\|^2$!

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: Suppose a point $\mathbf{x} \in \mathbb{R}^4$ has Euclidean norm $\|\mathbf{x}\| = 10$. Its scalar projections onto two orthogonal unit vectors $\mathbf{u}_1$ and $\mathbf{u}_2$ are $z_1 = 8$ and $z_2 = -5$. What is its squared reconstruction error $e^2$ when projected onto the 2D plane spanned by $\mathbf{u}_1$ and $\mathbf{u}_2$?

Answer: Using the multi-axis Pythagorean decomposition, $e^2 = \|\mathbf{x}\|^2 - (z_1^2 + z_2^2) = 10^2 - (8^2 + (-5)^2) = 100 - (64 + 25) = 100 - 89 = 11$.

• Orthogonality Guarantee: $\mathbf{u}^T(\mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}) = 0$ holds in any dimension $\mathbb{R}^d$ whenever $\mathbf{u}^T\mathbf{u} = 1$.
• Cross-Term Collapse: Expanding $\|\mathbf{x}_i - z_i\mathbf{u}\|^2$ yields $\|\mathbf{x}_i\|^2 - z_i^2$, turning a vector norm minimization into a scalar projection maximization.
• Canonical References: See Wikipedia Reference on the Pythagorean Theorem in Inner Product Spaces.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 2 Overview: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance
• Microlearning 2.1: 2.1 Total Data Energy as an Invariant Sum of Squared Norms
• Microlearning 2.2: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error
• Microlearning 2.3: 2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread
• Previous Lesson: 2.1 Total Data Energy as an Invariant Sum of Squared Norms
• Next Lesson: 2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread