Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD

Having unified the top-down and bottom-up foundations of Principal Component Analysis, we now turn theory into complete computational mastery. How do you execute PCA from start to finish on a raw table of numbers by hand? How do you decide whether to keep $k = 2$ or $k = 10$ principal components? And why do industrial libraries like scikit-learn, PyTorch, and LAPACK compute PCA using Singular Value Decomposition (SVD) directly on $X$ instead of explicitly forming $X^T X$?

In this final chapter, we walk through the complete end-to-end PCA pipeline—from centering and standardization to trace conservation, explained variance ratios, low-rank reconstruction, and numerical linear algebra via SVD.

In this Article

• 1. The Complete 5-Step PCA Algorithm from Raw Data to Reconstruction
• 2. End-to-End Numerical PCA by Hand on a 4-Point 2D Dataset
• 3. Trace Conservation and the Explained Variance Ratio
• 4. Covariance PCA vs. Standardized (Correlation) PCA
• 5. Why Production PCA Uses SVD (X = UΣVᵀ) Instead of Forming XᵀX
• 6. When PCA Fails: Non-Linear Manifolds and Variance ≠ Importance
• 7. Active Recall Check and Key Takeaways

1. The Complete 5-Step PCA Algorithm from Raw Data to Reconstruction

   [ Raw Data X_raw (n x d) ]
              |
              v  Step 1: Subtract column mean mu (and optionally divide by std dev s_j)
   [ Centered Data X (n x d) ]
              |
              v  Step 2: Compute SVD X = U Sigma V^T  (or Covariance C = (1/n) X^T X)
   [ Orthonormal Principal Axes V_k = [u_1, ..., u_k] (d x k) ]
              |
              v  Step 3: Project to low-D scores Z_k = X V_k (n x k)
   [ Compressed Representation Z_k ]
              |
              v  Step 4: Reconstruct in original space X_hat = Z_k V_k^T + mu^T
   [ Denoised / Compressed Approximation X_hat (n x d) ]

2. End-to-End Numerical PCA by Hand on a 4-Point 2D Dataset

Suppose we measure $n = 4$ observations on $d = 2$ features:

$$X_{\text{raw}} = \begin{bmatrix} 4 & 3 \\ 2 & 2 \\ 0 & 1 \\ -2 & -2 \end{bmatrix}$$

1. Step 1 — Center the Columns: The column means are $\mu_1 = \frac{4+2+0-2}{4} = 1$ and $\mu_2 = \frac{3+2+1-2}{4} = 1$. Subtracting $\boldsymbol{\mu} = (1, 1)^T$ from every row gives:

   $$X = \begin{bmatrix} 3 & 2 \\ 1 & 1 \\ -1 & 0 \\ -3 & -3 \end{bmatrix}$$

2. Step 2 — Form the Sample Covariance Matrix $C = \frac{1}{4}X^T X$:

   $$C = \frac{1}{4}\begin{bmatrix} 3^2+1^2+(-1)^2+(-3)^2 & 3(2)+1(1)+(-1)(0)+(-3)(-3) \\ 16 & 2^2+1^2+0^2+(-3)^2 \end{bmatrix} = \frac{1}{4}\begin{bmatrix} 20 & 16 \\ 16 & 14 \end{bmatrix} = \begin{bmatrix} 5.0 & 4.0 \\ 4.0 & 3.5 \end{bmatrix}$$

3. Step 3 — Solve the Characteristic Polynomial $\det(C - \lambda I) = 0$:
   • Trace: $\text{tr}(C) = 5.0 + 3.5 = 8.5$.
   • Determinant: $\det(C) = (5.0)(3.5) - (4.0)^2 = 17.5 - 16.0 = 1.5$.
   • Quadratic: $\lambda^2 - 8.5\lambda + 1.5 = 0 \implies \lambda = \frac{8.5 \pm \sqrt{72.25 - 6.0}}{2} = \frac{8.5 \pm \sqrt{66.25}}{2}$.
   • Eigenvalues: $\lambda_1 \approx 8.320$ and $\lambda_2 \approx 0.180$.
4. Step 4 — Find Principal Component 1 ($\mathbf{u}_1$) and Explained Variance Ratio:
   • From $(C - \lambda_1 I)\mathbf{u}_1 = \mathbf{0}$, we get $\mathbf{u}_1 \approx \begin{bmatrix} 0.769 \\ 0.639 \end{bmatrix}$.
   • Explained Variance Ratio of PC1: $\frac{\lambda_1}{\lambda_1 + \lambda_2} = \frac{8.320}{8.500} \approx 97.88\%$! Over $97.8\%$ of the 2D scatter cloud's variance is captured by a single 1D coordinate $z_i = 0.769 x_{i1} + 0.639 x_{i2}$.

3. Trace Conservation and the Explained Variance Ratio

Because rotating axes preserves the trace of a matrix, $\text{tr}(C) = \text{tr}(Q\Lambda Q^T) = \text{tr}(\Lambda)$, the sum of the original feature variances always equals the sum of all eigenvalues:

4. Covariance PCA vs. Standardized (Correlation) PCA

Because PCA maximizes variance, it is sensitive to measurement units. If Feature 1 is salary in dollars ($\sigma \approx 20,000$) and Feature 2 is years of experience ($\sigma \approx 5$), Feature 1 has $1.6 \times 10^7$ times larger variance and will completely dominate PC1! Whenever features have incomparable units or scales, standardize each column to zero mean and unit variance ($z\text{-score}$) before PCA—which is equivalent to computing the eigenvectors of the correlation matrix $R$.

5. Why Production PCA Uses SVD (X = UΣVᵀ) Instead of Forming XᵀX

Every centered data matrix $X \in \mathbb{R}^{n \times d}$ has a Singular Value Decomposition $X = U \Sigma V^T$, where $U \in \mathbb{R}^{n \times r}$ and $V \in \mathbb{R}^{d \times r}$ have orthonormal columns and $\Sigma = \text{diag}(\sigma_1, \dots, \sigma_r)$ contains the non-negative singular values. Plugging $X = U \Sigma V^T$ into $C = \frac{1}{n}X^T X$ gives:

$$C = \frac{1}{n} (V \Sigma^T U^T)(U \Sigma V^T) = V \left(\frac{\Sigma^2}{n}\right) V^T \implies \mathbf{u}_k = \mathbf{v}_k, \quad \lambda_k = \frac{\sigma_k^2}{n}$$

Quantity	Covariance Eigendecomposition ($C = \frac{1}{n}X^T X$)	Direct SVD on Centered Data ($X = U \Sigma V^T$)
Principal Directions	Eigenvectors $\mathbf{u}_k$ of $C$	Right singular vectors $\mathbf{v}_k$ (columns of $V$)
Eigenvalues (Variances)	Eigenvalues $\lambda_k$ of $C$	Squared singular values divided by $n$: $\lambda_k = \sigma_k^2 / n$
Principal Scores $Z$	$$Z = X Q$$	$$Z = X V = U \Sigma$$
Condition Number	$\kappa(X^T X) = \kappa(X)^2$ (Squares numerical error!)	$\kappa(X)$ (Preserves floating-point precision!)

6. When PCA Fails: Non-Linear Manifolds and Variance ≠ Importance

1. Non-Linear Structure: PCA only projects onto flat hyperplanes. If points lie on a spiral ("Swiss roll") or concentric spheres, linear projection scrambles them; use Kernel PCA, UMAP, or autoencoders instead.
2. Unsupervised Blindness to Class Labels: PCA is completely unsupervised. If two classes are separated along a low-variance direction while high-variance background noise runs perpendicular to the class boundary, dropping low-variance components destroys the classification signal (where Linear Discriminant Analysis, LDA, excels).

7. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: A centered data matrix $X$ with $n = 100$ rows has singular values $\sigma_1 = 30$, $\sigma_2 = 20$, and $\sigma_3 = 10$. What are the eigenvalues $\lambda_1, \lambda_2, \lambda_3$ of $C = \frac{1}{n}X^T X$ and what is the explained variance ratio of PC1?

Answer: Using $\lambda_k = \sigma_k^2 / n$, we get $\lambda_1 = 900/100 = 9$, $\lambda_2 = 400/100 = 4$, and $\lambda_3 = 100/100 = 1$. The total variance is $9 + 4 + 1 = 14$, so PC1 explains $9/14 \approx 64.29\%$ of the variance.

• Chapter Microlearnings: Complete your mastery with 6.1 Step-by-Step Numerical PCA by Hand, 6.2 Explained Variance Ratio & Trace Conservation, and 6.3 Why Production PCA Uses SVD.
• Canonical References: See Scikit-Learn Official PCA Documentation and Cross Validated on the Exact Relationship Between SVD and PCA.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 6 Overview: Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD
• Microlearning 6.1: 6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset
• Microlearning 6.2: 6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components
• Microlearning 6.3: 6.3 Why Production PCA Uses Singular Value Decomposition (SVD) Instead of Forming XᵀX
• Previous Lesson: 5.3 Why the Eigenvalue λ Equals the Exact Projected Variance Along Principal Component u
• Next Lesson: 6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset