2.1 Total Data Energy as an Invariant Sum of Squared Norms

Before we can understand how Principal Component Analysis redistributes variance across principal axes, we must establish a bedrock invariant: the total scatter (or total energy) of a centered dataset is a geometric constant. Whether you measure your data along the original coordinate axes or rotate your coordinate frame to any new orthonormal basis, the sum of the feature variances never changes.

In this Article

• 1. Defining Total Data Scatter via Squared Euclidean Norms
• 2. Why Average Squared Distance Equals the Sum of Feature Variances
• 3. Proof of Rotational Invariance Under Orthogonal Matrices
• 4. Worked Numerical Verification
• 5. Active Recall Check and Key Takeaways

1. Defining Total Data Scatter via Squared Euclidean Norms

For a centered dataset $\mathbf{x}_1, \dots, \mathbf{x}_n \in \mathbb{R}^d$ (where $\frac{1}{n}\sum_{i=1}^n \mathbf{x}_i = \mathbf{0}$), the squared Euclidean distance of observation $i$ from the origin is:

$$\|\mathbf{x}_i\|^2 = \mathbf{x}_i^T \mathbf{x}_i = \sum_{j=1}^d x_{ij}^2$$

Averaging this squared distance across all $n$ observations defines the Total Data Scatter (often called total variance or total signal energy):

$$E_{\text{total}} = \frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i\|^2 = \frac{1}{n}\|X\|_F^2$$

where $\|X\|_F = \sqrt{\sum_{i=1}^n \sum_{j=1}^d x_{ij}^2}$ is the Frobenius norm of the centered data matrix $X$.

2. Why Average Squared Distance Equals the Sum of Feature Variances

Notice what happens when we swap the order of summation between observations $i$ and features $j$:

$$E_{\text{total}} = \frac{1}{n}\sum_{i=1}^n \left(\sum_{j=1}^d x_{ij}^2\right) = \sum_{j=1}^d \underbrace{\left(\frac{1}{n}\sum_{i=1}^n x_{ij}^2\right)}_{\text{Var}(x_j)} = \sum_{j=1}^d \text{Var}(x_j)$$

3. Proof of Rotational Invariance Under Orthogonal Matrices

Suppose we rotate our coordinate axes using any $d \times d$ orthogonal matrix $Q$ (whose columns are mutually perpendicular unit vectors, so $Q^T Q = I_d$). In the rotated coordinate system, observation $\mathbf{x}_i$ has new coordinates $\mathbf{z}_i = Q^T \mathbf{x}_i$. What is its new squared distance from the origin?

$$\|\mathbf{z}_i\|^2 = \mathbf{z}_i^T \mathbf{z}_i = (Q^T \mathbf{x}_i)^T (Q^T \mathbf{x}_i) = \mathbf{x}_i^T (Q Q^T) \mathbf{x}_i = \mathbf{x}_i^T I_d \mathbf{x}_i = \|\mathbf{x}_i\|^2$$

Because every single point preserves its Euclidean norm ($\|\mathbf{z}_i\|^2 = \|\mathbf{x}_i\|^2$), the sum of variances across all $d$ rotated axes equals the sum of variances across the original axes: $\sum_{j=1}^d \text{Var}(z_j) = \sum_{j=1}^d \text{Var}(x_j) = E_{\text{total}}$. Rotation never creates or destroys variance—it only redistributes variance among the axes!

4. Worked Numerical Verification

Consider two centered points $\mathbf{x}_1 = \begin{bmatrix} 3 \\ 1 \end{bmatrix}$ and $\mathbf{x}_2 = \begin{bmatrix} -3 \\ -1 \end{bmatrix}$.

• Original Variances: $\text{Var}(x_1) = \frac{3^2 + (-3)^2}{2} = 9$, and $\text{Var}(x_2) = \frac{1^2 + (-1)^2}{2} = 1$. Total scatter: $E_{\text{total}} = 9 + 1 = 10$.
• Rotate by $45^\circ$: Using $Q = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}$, the new coordinates are $\mathbf{z}_1 = \frac{1}{\sqrt{2}}\begin{bmatrix} 3+1 \\ -3+1 \end{bmatrix} = \begin{bmatrix} 2\sqrt{2} \\ -\sqrt{2} \end{bmatrix}$ and $\mathbf{z}_2 = \begin{bmatrix} -2\sqrt{2} \\ \sqrt{2} \end{bmatrix}$.
• Rotated Variances: $\text{Var}(z_1) = (2\sqrt{2})^2 = 8$, and $\text{Var}(z_2) = (-\sqrt{2})^2 = 2$. Their sum is $8 + 2 = 10 = E_{\text{total}}$!

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: A centered 2D dataset has $\text{Var}(x_1) = 14$ and $\text{Var}(x_2) = 6$. After rotating the coordinate system to the principal component axes $(\mathbf{u}_1, \mathbf{u}_2)$, you find that the variance along PC1 is $18$. What must be the variance along the perpendicular axis PC2?

Answer: Because orthogonal rotation preserves total scatter, $\text{Var}(z_1) + \text{Var}(z_2) = \text{Var}(x_1) + \text{Var}(x_2) = 14 + 6 = 20$. Therefore, $\text{Var}(z_2) = 20 - 18 = 2$.

• Total Scatter Conservation: $E_{\text{total}} = \frac{1}{n}\sum_{i=1}^n \|\mathbf{x}_i\|^2 = \sum_{j=1}^d \text{Var}(x_j)$ is invariant under any orthogonal rotation of the axes.
• PCA as Variance Concentration: Since total variance is fixed, PCA simply rotates the axes to pack as much of that fixed budget as possible into the first few coordinates.
• Canonical References: See Wikipedia Reference on Orthogonal Transformations and Frobenius Norm Invariance.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 2 Overview: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance
• Microlearning 2.1: 2.1 Total Data Energy as an Invariant Sum of Squared Norms
• Microlearning 2.2: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error
• Microlearning 2.3: 2.3 Why Minimizing Orthogonal Error Is Identical to Maximizing Projected Spread
• Previous Lesson: Chapter 2: The Pythagorean Miracle — How Minimizing Reconstruction Error Forces You to Maximize Variance
• Next Lesson: 2.2 Deriving the Pythagorean Decomposition of Data, Shadow, and Error