1.3 Orthogonal Reconstruction Error vs. Vertical Regression Residuals

A frequent point of confusion for students learning Principal Component Analysis is: "Don't we already have Ordinary Least Squares (OLS) linear regression to fit a line through a cloud of points? Why does PCA produce a different line?"

The difference is geometric and fundamental. OLS regression assumes one coordinate ($x$) is a noise-free input and the other ($y$) is a noisy target, so it minimizes vertical errors parallel to the $y$-axis. PCA treats all coordinates symmetrically as noisy measurements of an underlying latent phenomenon, minimizing orthogonal (perpendicular) distances from each point to the line.

In this Article

• 1. Visualizing Vertical OLS Residuals vs. Orthogonal PCA Errors
• 2. Mathematical Formulation of Both Error Objectives
• 3. The Axis-Swap Paradox in Regression vs. Rotational Invariance in PCA
• 4. Concrete Worked Comparison on a Symmetric Cloud
• 5. Active Recall Check and Key Takeaways

1. Visualizing Vertical OLS Residuals vs. Orthogonal PCA Errors

Imagine a point $\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \end{bmatrix}$ sitting above a candidate line passing through the origin with slope $m$. There are three distinct ways to measure the gap between the point and the line:

1. Vertical Residual (Regressing $x_2$ on $x_1$): Hold $x_{i1}$ fixed and drop straight down to $\hat{x}_{i2} = m x_{i1}$.
2. Horizontal Residual (Regressing $x_1$ on $x_2$): Hold $x_{i2}$ fixed and move horizontally to $\hat{x}_{i1} = x_{i2}/m$.
3. Orthogonal Residual (PCA / Total Least Squares): Move along the shortest perpendicular path to $\hat{\mathbf{x}}_i = (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$.

            * x_i = (x_i1, x_i2)
            |  \
   Vertical |   \  Orthogonal PCA Residual e_i
   OLS      |    \ (Shortest 90-degree path to line)
   Residual |     v
   |y - mx| |     (o) x_hat_i = (u^T x_i) u
            v    /
           [x]  /  <-- Candidate Line (slope m)
               /
       Origin 0

2. Mathematical Formulation of Both Error Objectives

For a line of slope $m$ passing through the origin, the unit direction vector is $\mathbf{u} = \frac{1}{\sqrt{1+m^2}}\begin{bmatrix} 1 \\ m \end{bmatrix}$, and the unit normal vector perpendicular to the line is $\mathbf{n} = \frac{1}{\sqrt{1+m^2}}\begin{bmatrix} -m \\ 1 \end{bmatrix}$.

• OLS Vertical Squared Error: $E_{\text{OLS}}(m) = \frac{1}{n}\sum_{i=1}^n (x_{i2} - m x_{i1})^2$
• PCA Orthogonal Squared Error: Because $e_i = |\mathbf{n} \cdot \mathbf{x}_i|$, the perpendicular squared distance is:

  $$E_{\text{PCA}}(m) = \frac{1}{n}\sum_{i=1}^n (\mathbf{n}^T \mathbf{x}_i)^2 = \frac{1}{n}\sum_{i=1}^n \frac{(x_{i2} - m x_{i1})^2}{1 + m^2}$$

3. The Axis-Swap Paradox in Regression vs. Rotational Invariance in PCA

In OLS regression, the optimal slope for predicting $x_2$ from $x_1$ is $m_{2|1} = \frac{\text{Cov}(x_1, x_2)}{\text{Var}(x_1)} = r \frac{s_2}{s_1}$, where $r$ is the Pearson correlation. If you swap the axes and regress $x_1$ on $x_2$, the resulting line has slope $1 / m_{1|2} = \frac{1}{r} \frac{s_2}{s_1}$ in the original plot!

Unless the data lies on a perfect straight line ($|r| = 1$), OLS gives two different lines depending on which variable you label as the horizontal axis. By contrast, PCA's orthogonal error $\|\mathbf{x}_i - (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}\|^2$ depends only on Euclidean lengths and angles, so rotating or swapping the axes rotates the principal component line by the exact same angle.

4. Concrete Worked Comparison on a Symmetric Cloud

Consider four centered points forming a diamond tilted along $45^\circ$: $\mathbf{x}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$, $\mathbf{x}_2 = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$, $\mathbf{x}_3 = \begin{bmatrix} -1 \\ -2 \end{bmatrix}$, $\mathbf{x}_4 = \begin{bmatrix} -2 \\ -1 \end{bmatrix}$.

Method	Formula Applied	Resulting Line Slope $m$
OLS: $x_2$ on $x_1$	$$m = \frac{\sum x_{i1} x_{i2}}{\sum x_{i1}^2} = \frac{2+2+2+2}{4+1+1+4} = \frac{8}{10}$$	$m = 0.80$ (Flattened below $45^\circ$!)
OLS: $x_1$ on $x_2$	$$x_1 = 0.80 x_2 \implies x_2 = \frac{1}{0.80} x_1$$	$m = 1.25$ (Steepened above $45^\circ$!)
PCA (Orthogonal)	Minimizes $\sum \frac{(x_{i2} - m x_{i1})^2}{1+m^2}$	$m = 1.00$ (Exact $45^\circ$ symmetry axis!)

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: For the point $\mathbf{x} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$ and the line $x_2 = 2x_1$ (slope $m=2$), calculate both the squared vertical OLS residual and the squared orthogonal PCA residual.

Answer: The vertical prediction at $x_1 = 1$ is $\hat{x}_2 = 2(1) = 2$, so the squared vertical residual is $(3 - 2)^2 = 1.0$. The squared orthogonal PCA residual divides by $1 + m^2 = 1 + 2^2 = 5$: $e_{\text{PCA}}^2 = \frac{(3 - 2(1))^2}{1 + 2^2} = \frac{1}{5} = 0.2$. You can verify this via projection onto $\mathbf{u} = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ 2 \end{bmatrix}$: $z = \frac{1+6}{\sqrt{5}} = \frac{7}{\sqrt{5}}$, $\hat{\mathbf{x}} = \begin{bmatrix} 1.4 \\ 2.8 \end{bmatrix}$, and $\|\mathbf{x} - \hat{\mathbf{x}}\|^2 = (-0.4)^2 + (0.2)^2 = 0.16 + 0.04 = 0.20$.

• OLS is for asymmetric prediction of a designated target $y$ from a predictor $x$; PCA is for symmetric representation of all coordinates together.
• Orthogonal distance introduces the factor $\frac{1}{1+m^2}$, making PCA invariant to coordinate rotations and axis permutations.
• Canonical References: See Wikipedia Reference on Total Least Squares and Orthogonal Regression and CrossValidated Discussion on Orthogonal Projections vs. Regression.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 1 Overview: Chapter 1: The Geometry of Data Compression — Centering, Projections, and Orthogonal Shadows
• Microlearning 1.1: 1.1 Centering Multivariate Data and Shifting the Origin to the Mean
• Microlearning 1.2: 1.2 Unit Direction Vectors and Scalar Dot-Product Projections
• Microlearning 1.3: 1.3 Orthogonal Reconstruction Error vs. Vertical Regression Residuals
• Previous Lesson: 1.2 Unit Direction Vectors and Scalar Dot-Product Projections
• Next Lesson: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance