Chapter 1: The Geometry of Data Compression — Centering, Projections, and Orthogonal Shadows

Principal Component Analysis (PCA) begins with a concrete engineering question: how do you compress a high-dimensional cloud of observations onto a lower-dimensional subspace while losing as little geometric structure as possible? Before a single matrix determinant or eigenvalue ever enters the picture, PCA is fundamentally a shadow-casting problem in Euclidean space.

When we approach PCA from the top down, we start with raw data points, shift their center of gravity to the origin, project every point perpendicularly onto a candidate axis $\mathbf{u}$, and measure the distortion caused by flattening the cloud. Understanding these three geometric operations—centering, orthogonal projection, and perpendicular residuals—unlocks the entire logic of dimensionality reduction.

In this Article

• 1. The Top-Down Problem: Compressing 2D Data onto a 1D Line
• 2. Step 1: Mean-Centering Shifts the Origin to the Cloud Centroid
• 3. Step 2: Casting Perpendicular Shadows with Unit Vectors
• 4. Step 3: Orthogonal Reconstruction Error vs. Regression Residuals
• 5. Complete Worked Numerical Example
• 6. Spatial Analogy and Where It Breaks Down
• 7. Critical Misconceptions and Edge Cases
• 8. Active Recall Check and Key Takeaways

1. The Top-Down Problem: Compressing 2D Data onto a 1D Line

Suppose you record two correlated measurements—say height $x_{i1}$ and arm span $x_{i2}$—for $n$ individuals. Each person is represented as a column vector $\mathbf{x}_i = \begin{bmatrix} x_{i1} \\ x_{i2} \end{bmatrix}$ in $\mathbb{R}^2$. Because height and arm span grow together, the scatter plot forms a tilted elliptical cloud rather than a uniform square.

If storage or downstream modeling requires compressing those two numbers per person into a single summary score $z_i \in \mathbb{R}$, we must choose a straight line passing through the center of the cloud and replace each 2D point $\mathbf{x}_i$ with its closest point $\hat{\mathbf{x}}_i$ on that line.

      x2 (Arm Span, Centered)
       ^
       |                * x_i (True 2D Data Point)
       |               / \
       |              /   \  e_i (Perpendicular Error)
       |             /     v
       |      *     /      (o) x_hat_i = (x_i . u) u  [Shadow on Line]
       |           /      /
-------+----------0------+-----------------> x1 (Height, Centered)
       |         /      /
       |   (o)  /      /   <-- Candidate Line spanned by unit vector u
       |    ^  /
       |     \/
       |     *

2. Step 1: Mean-Centering Shifts the Origin to the Cloud Centroid

A linear subspace (a line or plane in linear algebra) must pass through the coordinate origin $\mathbf{0}$. If our raw data cloud is centered around $\boldsymbol{\mu} = \begin{bmatrix} 170 \\ 172 \end{bmatrix}$ centimeters, any line forced through $\begin{bmatrix} 0 \\ 0 \end{bmatrix}$ would primarily point from the origin toward the average human size rather than capturing person-to-person variation.

To isolate internal variation, we compute the sample mean vector $\boldsymbol{\mu} = \frac{1}{n}\sum_{i=1}^n \mathbf{x}_i^{\text{raw}}$ and subtract it from every observation:

$$\mathbf{x}_i = \mathbf{x}_i^{\text{raw}} - \boldsymbol{\mu}, \quad \text{so that} \quad \frac{1}{n}\sum_{i=1}^n \mathbf{x}_i = \mathbf{0}$$

This translation preserves every pairwise distance and angle between observations while placing the centroid of the cloud directly at $\mathbf{0}$.

3. Step 2: Casting Perpendicular Shadows with Unit Vectors

Once the cloud is centered at $\mathbf{0}$, any candidate line through the origin is defined by a unit direction vector $\mathbf{u} \in \mathbb{R}^d$ satisfying $\|\mathbf{u}\| = \sqrt{\mathbf{u}^T \mathbf{u}} = 1$. Restricting $\mathbf{u}$ to unit length ensures that $\mathbf{u}$ acts purely as a directional compass without stretching or shrinking our measurement scale.

4. Step 3: Orthogonal Reconstruction Error vs. Regression Residuals

When we approximate $\mathbf{x}_i$ by its shadow $\hat{\mathbf{x}}_i$, the residual vector is $\mathbf{r}_i = \mathbf{x}_i - \hat{\mathbf{x}}_i$, and its Euclidean length is the orthogonal reconstruction error $e_i = \|\mathbf{x}_i - (\mathbf{x}_i \cdot \mathbf{u})\mathbf{u}\|$.

Property	Ordinary Least Squares (OLS Regression)	Principal Component Analysis (PCA)
Variable Roles	Asymmetric: $x_1$ is noise-free predictor, $x_2$ is response	Symmetric: all features $x_1, x_2, \dots, x_d$ treated equally
Error Direction	Vertical distance parallel to the $x_2$ axis	Perpendicular (orthogonal) distance to the line $\mathbf{u}$
Rotation Invariance	Not rotation-invariant; swapping axes changes the line	Fully rotation-invariant; geometry depends only on point cloud

5. Complete Worked Numerical Example

Consider a raw 2D dataset of $n = 2$ points: $\mathbf{x}_1^{\text{raw}} = \begin{bmatrix} 5 \\ 4 \end{bmatrix}$ and $\mathbf{x}_2^{\text{raw}} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$.

1. Center the data: The mean vector is $\boldsymbol{\mu} = \frac{1}{2}\begin{bmatrix} 5+1 \\ 4+2 \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \end{bmatrix}$. Subtracting $\boldsymbol{\mu}$ gives $\mathbf{x}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$ and $\mathbf{x}_2 = \begin{bmatrix} -2 \\ -1 \end{bmatrix}$.
2. Project onto a candidate axis: Let us test the horizontal unit axis $\mathbf{u} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$. The 1D score for $\mathbf{x}_1$ is $z_1 = \begin{bmatrix} 2 & 1 \end{bmatrix}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = 2$.
3. Reconstruct and measure error: The shadow on the horizontal axis is $\hat{\mathbf{x}}_1 = 2\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 2 \\ 0 \end{bmatrix}$. The residual vector is $\mathbf{x}_1 - \hat{\mathbf{x}}_1 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$, giving squared error $e_1^2 = 0^2 + 1^2 = 1$.
4. Compare with the optimal axis: If instead we choose $\mathbf{u}^* = \begin{bmatrix} 2/\sqrt{5} \\ 1/\sqrt{5} \end{bmatrix}$, then $z_1 = \frac{4+1}{\sqrt{5}} = \sqrt{5}$, the reconstructed point is $\hat{\mathbf{x}}_1 = \sqrt{5}\begin{bmatrix} 2/\sqrt{5} \\ 1/\sqrt{5} \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$, and the reconstruction error drops to $e_1^2 = 0$!

6. Spatial Analogy and Where It Breaks Down

Think of the 2D data cloud as a constellation of glowing fireflies frozen in a dark room, and the unit vector $\mathbf{u}$ as a straight metal rod passing through the center of the swarm. Projecting each point onto $\mathbf{u}$ is like connecting each firefly to the rod with the shortest possible perpendicular elastic spring; the best rod orientation minimizes the total elastic energy stored in all springs.

Where the analogy breaks down: Physical springs exert mechanical forces that pull the rod according to torque around a pivot, whereas in PCA every spring length is squared ($e_i^2$) precisely so that orthogonal components decouple cleanly via the Pythagorean theorem across arbitrary dimensions $\mathbb{R}^d$.

7. Critical Misconceptions and Edge Cases

8. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: A centered 2D point is $\mathbf{x} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$, and our candidate axis is at $45^\circ$ with unit vector $\mathbf{u} = \begin{bmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix}$. What is the scalar projection $z$, the 2D shadow $\hat{\mathbf{x}}$, and the squared orthogonal error $e^2$?

Answer: First, compute the dot product: $z = \mathbf{x} \cdot \mathbf{u} = \frac{3 + 4}{\sqrt{2}} = \frac{7}{\sqrt{2}}$. Next, multiply $z$ by $\mathbf{u}$ to get the 2D shadow: $\hat{\mathbf{x}} = \frac{7}{\sqrt{2}} \begin{bmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix} = \begin{bmatrix} 3.5 \\ 3.5 \end{bmatrix}$. Finally, the residual vector is $\mathbf{x} - \hat{\mathbf{x}} = \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix}$, so the squared error is $e^2 = (-0.5)^2 + 0.5^2 = 0.5$.

• Mean-centering places the dataset's center of mass at the origin $\mathbf{0}$ so linear subspaces capture internal spread rather than distance from the origin.
• Unit direction vectors ($\|\mathbf{u}\|=1$) allow scalar projections to be written simply as $z_i = \mathbf{u}^T\mathbf{x}_i$ and vector reconstructions as $\hat{\mathbf{x}}_i = (\mathbf{u}^T\mathbf{x}_i)\mathbf{u}$.
• Orthogonal residuals treat all features symmetrically, distinguishing PCA from vertical least-squares regression.
• Canonical References: Explore Jonathan Shlens' Tutorial on Principal Component Analysis (arXiv:1404.1100) and the Wikipedia Reference on Principal Component Analysis.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 1 Overview: Chapter 1: The Geometry of Data Compression — Centering, Projections, and Orthogonal Shadows
• Microlearning 1.1: 1.1 Centering Multivariate Data and Shifting the Origin to the Mean
• Microlearning 1.2: 1.2 Unit Direction Vectors and Scalar Dot-Product Projections
• Microlearning 1.3: 1.3 Orthogonal Reconstruction Error vs. Vertical Regression Residuals
• Next Lesson: 1.1 Centering Multivariate Data and Shifting the Origin to the Mean