6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset

Nothing solidifies linear algebra like executing every step of Principal Component Analysis by hand on clean numbers where you can track every coordinate from raw input to compressed 1D score and back to 2D reconstruction.

In this microlearning, we work through a complete numerical example chosen so that all arithmetic—mean-centering, covariance construction, characteristic roots, unit eigenvectors, 1D projection scores, and orthogonal reconstruction errors—comes out to exact, transparent expressions.

In this Article

• 1. The Raw 4-Point Dataset and Step 1: Mean-Centering
• 2. Step 2 & 3: Covariance Matrix C and Exact Eigenpairs
• 3. Step 4 & 5: 1D Principal Scores zᵢ and 2D Reconstructed Points
• 4. Verifying the Pythagorean Accounting Identity to the Last Decimal
• 5. Active Recall Check and Key Takeaways

1. The Raw 4-Point Dataset and Step 1: Mean-Centering

Consider $n = 4$ observations of two features $(x_1, x_2)$:

$$\mathbf{x}_1^{\text{raw}} = (6, 5), \quad \mathbf{x}_2^{\text{raw}} = (5, 6), \quad \mathbf{x}_3^{\text{raw}} = (2, 1), \quad \mathbf{x}_4^{\text{raw}} = (3, 0)$$

The sample mean vector is $\boldsymbol{\mu} = \left(\frac{6+5+2+3}{4}, \frac{5+6+1+0}{4}\right) = (4, 3)$. Subtracting $\boldsymbol{\mu} = (4, 3)$ from each observation gives the centered matrix $X$:

$$X = \begin{bmatrix} 2 & 2 \\ 1 & 3 \\ -2 & -2 \\ -1 & -3 \end{bmatrix}$$

2. Step 2 & 3: Covariance Matrix C and Exact Eigenpairs

We compute $C = \frac{1}{4}X^T X$:

• $$c_{11} = \frac{1}{4}(4 + 1 + 4 + 1) = \frac{10}{4} = 2.5$$
• $$c_{22} = \frac{1}{4}(4 + 9 + 4 + 9) = \frac{26}{4} = 6.5$$
• $$c_{12} = \frac{1}{4}(4 + 3 + 4 + 3) = \frac{14}{4} = 3.5$$

Thus $C = \begin{bmatrix} 2.5 & 3.5 \\ 3.5 & 6.5 \end{bmatrix}$. Now solve $\det(C - \lambda I) = 0$:

• $$\text{tr}(C) = 2.5 + 6.5 = 9.0$$
• $$\det(C) = (2.5)(6.5) - (3.5)^2 = 16.25 - 12.25 = 4.0$$
• Characteristic equation: $\lambda^2 - 9\lambda + 4 = 0 \implies \lambda_1 = \frac{9 + \sqrt{65}}{2} \approx 8.531$, $\lambda_2 = \frac{9 - \sqrt{65}}{2} \approx 0.469$.

For a symmetric $2 \times 2$ matrix with $c_{12} \neq 0$, the unit eigenvector for $\lambda_1$ is proportional to $\begin{bmatrix} c_{12} \\ \lambda_1 - c_{11} \end{bmatrix} = \begin{bmatrix} 3.5 \\ 6.031 \end{bmatrix}$, which normalizes to:

$$\mathbf{u}_1 \approx \begin{bmatrix} 0.502 \\ 0.865 \end{bmatrix}, \qquad \mathbf{u}_2 \approx \begin{bmatrix} -0.865 \\ 0.502 \end{bmatrix}$$

3. Step 4 & 5: 1D Principal Scores zᵢ and 2D Reconstructed Points

Projecting onto PC1 via $z_i = \mathbf{x}_i^T \mathbf{u}_1$ and reconstructing in raw coordinates via $\hat{\mathbf{x}}_i^{\text{raw}} = \boldsymbol{\mu} + z_i \mathbf{u}_1$ yields:

Point $i$	Centered $\mathbf{x}_i$	1D Score $z_i = \mathbf{x}_i \cdot \mathbf{u}_1$	Centered Shadow $\hat{\mathbf{x}}_i = z_i \mathbf{u}_1$	Squared Error $e_i^2 = \|\mathbf{x}_i - \hat{\mathbf{x}}_i\|^2$
1	$$(2, 2)$$	$$+2.734$$	$$(1.372, 2.365)$$	$$0.527$$
2	$$(1, 3)$$	$$+3.097$$	$$(1.555, 2.679)$$	$$0.411$$
3	$$(-2, -2)$$	$$-2.734$$	$$(-1.372, -2.365)$$	$$0.527$$
4	$$(-1, -3)$$	$$-3.097$$	$$(-1.555, -2.679)$$	$$0.411$$

4. Verifying the Pythagorean Accounting Identity to the Last Decimal

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: In the worked example above, why must we add $\boldsymbol{\mu} = (4, 3)$ back at the very end when reconstructing $\hat{\mathbf{x}}_i^{\text{raw}} = \boldsymbol{\mu} + z_i \mathbf{u}_1$?

Answer: Because $z_i \mathbf{u}_1$ reconstructs the point relative to the centered origin $(0, 0)$. Adding the mean vector $\boldsymbol{\mu}$ shifts the reconstructed shadow back to the original coordinate frame of the raw dataset.

• 2x2 Eigenvector Formula: For $C = \begin{bmatrix} c_{11} & c_{12} \\ c_{12} & c_{22} \end{bmatrix}$ with $c_{12} \neq 0$, the eigenvector for $\lambda_1$ points along $(c_{12}, \lambda_1 - c_{11})^T$.
• Exact Error Verification: The average squared reconstruction error when keeping PC1 equals $\lambda_2$ to the last decimal place.
• Canonical References: See Wikipedia Reference on Computing PCA via the Covariance Method.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 6 Overview: Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD
• Microlearning 6.1: 6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset
• Microlearning 6.2: 6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components
• Microlearning 6.3: 6.3 Why Production PCA Uses Singular Value Decomposition (SVD) Instead of Forming XᵀX
• Previous Lesson: Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD
• Next Lesson: 6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components