6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components

When you run PCA on a 100-dimensional dataset, it returns 100 orthogonal principal components ordered by their eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_{100} \ge 0$. If you keep all 100 components, you have merely rotated your coordinate system without compressing anything! How do you decide how many components $k$ to retain?

The mathematical foundation for choosing $k$ is trace conservation: because the trace of a matrix is invariant under orthogonal similarity transformations, the sum of all eigenvalues equals the total variance of the original features.

In this Article

• 1. Proving Trace Conservation: tr(C) = Σ Var(xⱼ) = Σ λⱼ
• 2. Explained Variance Ratio (EVR) and Scree Plots
• 3. Practical Rules for Selecting k in Machine Learning and Science
• 4. Worked 5D Scree Table and Reconstruction Error Calculation
• 5. Active Recall Check and Key Takeaways

1. Proving Trace Conservation: tr(C) = Σ Var(xⱼ) = Σ λⱼ

For any two matrices $A$ and $B$ of compatible dimensions, the cyclic property of the trace states that $\text{tr}(AB) = \text{tr}(BA)$. Applying this to the spectral decomposition $C = Q \Lambda Q^T$ (where $Q^T Q = I$) yields:

$$\text{tr}(C) = \text{tr}(Q \Lambda Q^T) = \text{tr}(\Lambda Q^T Q) = \text{tr}(\Lambda I) = \text{tr}(\Lambda) = \sum_{j=1}^d \lambda_j$$

Geometrically, rotating a rigid cloud of points neither creates nor destroys variance: it simply redistributes the total variance $\text{tr}(C)$ across the principal axes so that the leading axes pack as much variance as possible.

2. Explained Variance Ratio (EVR) and Scree Plots

   SCREE PLOT (Eigenvalue vs. Component Index k):
   lambda_k ^
       12.0 |  *  (PC1: Big signal)
        8.0 |     *  (PC2: Big signal)
        2.0 |        *  <-- "Elbow" at k = 2 or 3
        0.5 |           *   *   *   *   *   (Noise floor: flat tail of tiny eigenvalues)
            +-------------------------------> Component k
               1  2  3  4   5   6   7   8

3. Practical Rules for Selecting k in Machine Learning and Science

1. Cumulative Variance Threshold (e.g., $90\%$, $95\%$, or $99\%$): Choose the smallest $k$ such that $\text{CEVR}(k) \ge 0.95$. In scikit-learn, passing a float PCA(n_components=0.95) automatically selects this $k$.
2. Scree Plot Elbow (Cattell's Criterion): Plot $\lambda_k$ versus $k$ and truncate where the steep cliff of signal eigenvalues transitions into a flat rubble ("scree") of background noise.
3. Kaiser Criterion (for Standardized Correlation PCA): When all features are standardized to variance $1.0$ (so $\text{tr}(R) = d$ and the average eigenvalue is $1.0$), retain components with $\lambda_k > 1.0$ (components that explain more variance than a single original variable).

4. Worked 5D Scree Table and Reconstruction Error Calculation

Suppose a 5D dataset has diagonal covariance entries $3.2, 4.8, 2.5, 5.5, 4.0$ (so $\text{tr}(C) = 20.0$), and its eigenvalues are $\lambda_1 = 13.0, \lambda_2 = 5.0, \lambda_3 = 1.2, \lambda_4 = 0.5, \lambda_5 = 0.3$.

Component $k$	Eigenvalue $\lambda_k$	Individual $\text{EVR}_k$	Cumulative $\text{CEVR}(k)$	Relative Reconstruction Error
1	$$13.0$$	$$65.0\%$$	$$65.0\%$$	$$35.0\%$$
2	$$5.0$$	$$25.0\%$$	$$90.0\%$$	$$10.0\%$$
3	$$1.2$$	$$6.0\%$$	$$96.0\%$$	$$4.0\%$$
4	$$0.5$$	$$2.5\%$$	$$98.5\%$$	$$1.5\%$$
5	$$0.3$$	$$1.5\%$$	$$100.0\%$$	$$0.0\%$$

If our engineering target is $\ge 95\%$ variance retention (or $\le 5\%$ relative squared reconstruction error), we select $k = 3$ components, capturing $96.0\%$ of the variance while compressing storage by $40\%$.

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: You run PCA on a standardized correlation matrix of $d = 12$ features (so every feature has variance $1.0$). Without computing a single eigenvalue, what is the exact sum of all 12 eigenvalues?

Answer: By trace conservation, $\sum_{j=1}^{12} \lambda_j = \text{tr}(R) = 12 \times 1.0 = 12$.

• Trace Conservation: $\sum_{j=1}^d \lambda_j = \text{tr}(C)$ holds for every symmetric covariance matrix.
• Dual Meaning of $1 - \text{CEVR}(k)$: The uncaptured variance fraction $1 - \text{CEVR}(k)$ equals the relative mean squared reconstruction error.
• Canonical References: See Wikipedia Reference on Scree Plots.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 6 Overview: Chapter 6: PCA From Napkin Math to Production Code — Hand Calculations, Variance Ratios, and SVD
• Microlearning 6.1: 6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset
• Microlearning 6.2: 6.2 Explained Variance Ratio, Trace Conservation, and Choosing k Principal Components
• Microlearning 6.3: 6.3 Why Production PCA Uses Singular Value Decomposition (SVD) Instead of Forming XᵀX
• Previous Lesson: 6.1 Step-by-Step Numerical PCA by Hand on a 2D Dataset
• Next Lesson: 6.3 Why Production PCA Uses Singular Value Decomposition (SVD) Instead of Forming XᵀX