4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data

When working with multivariable datasets, writing out separate summation formulas for every feature's variance and every pair of features' covariance quickly becomes unmanageable. Linear algebra compresses all $d^2$ variance and covariance calculations into a single matrix product: $C = \frac{1}{n}X^T X$.

In this microlearning, we unpack both the column-inner-product view and the row-outer-product view of $\frac{1}{n}X^T X$, and clarify when to divide by $n$ versus $n - 1$.

In this Article

• 1. Layout of the Centered Data Matrix X
• 2. Two Views of XᵀX: Feature Inner Products vs. Sample Outer Products
• 3. Dividing by n vs. n − 1: Why PCA Eigenvectors Never Change
• 4. Worked Step-by-Step Matrix Construction
• 5. Active Recall Check and Key Takeaways

1. Layout of the Centered Data Matrix X

By standard statistical and machine learning convention (such as scikit-learn and NumPy), our centered dataset is stored as an $n \times d$ matrix $X$:

• Rows ($i = 1, \dots, n$): Individual mean-centered observations $\mathbf{x}_i^T \in \mathbb{R}^{1 \times d}$.
• Columns ($j = 1, \dots, d$): Mean-centered feature vectors $\mathbf{f}_j \in \mathbb{R}^{n \times 1}$, where the sum of entries in each column is zero.

            Feature 1   Feature 2  ...  Feature d
           [  x_11        x_12     ...    x_1d  ]  <-- Observation 1 (row x_1^T)
      X =  [  x_21        x_22     ...    x_2d  ]  <-- Observation 2 (row x_2^T)
           [   :           :       ...     :    ]
           [  x_n1        x_n2     ...    x_nd  ]  <-- Observation n (row x_n^T)
              (n x d matrix, column means = 0)

2. Two Algebraic Views of XᵀX: Feature Inner Products vs. Sample Outer Products

When we multiply $X^T$ (size $d \times n$) by $X$ (size $n \times d$), we obtain a $d \times d$ matrix $X^T X$ that can be read in two complementary ways:

3. Dividing by n vs. n − 1: Why PCA Eigenvectors Never Change

In statistics courses, sample covariance is often defined with Bessel's correction $C_{\text{unbiased}} = \frac{1}{n-1}X^T X$ so that $\mathbb{E}[C_{\text{unbiased}}] = \Sigma$ when the mean is estimated from the sample. In optimization derivations, $C_{\text{MLE}} = \frac{1}{n}X^T X$ is cleaner.

Does $n$ vs. $n-1$ change PCA? Not the principal directions! Scaling a matrix by a positive constant $\alpha = \frac{n}{n-1}$ scales every eigenvalue by $\alpha$ while leaving every unit eigenvector 100% identical, and even the explained variance ratios $\lambda_k / \sum_j \lambda_j$ remain completely invariant because the constant cancels out in numerator and denominator.

4. Worked Step-by-Step Matrix Construction

Let $X = \begin{bmatrix} 3 & 1 \\ -1 & -2 \\ -2 & 1 \end{bmatrix}$ ($n=3$ centered observations in $d=2$ dimensions; verify that each column sums to $0$).

1. Compute $X^T X$:

   $$X^T X = \begin{bmatrix} 3 & -1 & -2 \\ 1 & -2 & 1 \end{bmatrix} \begin{bmatrix} 3 & 1 \\ -1 & -2 \\ -2 & 1 \end{bmatrix} = \begin{bmatrix} 9+1+4 & 3+2-2 \\ 3+2-2 & 1+4+1 \end{bmatrix} = \begin{bmatrix} 14 & 3 \\ 3 & 6 \end{bmatrix}$$

2. Divide by $n = 3$ (or $n-1 = 2$):

   $$C = \frac{1}{3}\begin{bmatrix} 14 & 3 \\ 3 & 6 \end{bmatrix} = \begin{bmatrix} 4.67 & 1.00 \\ 1.00 & 2.00 \end{bmatrix}$$

5. Active Recall Check and Key Takeaways

Quick Check: Test Your Understanding

Question: If your dataset has $n = 10,000$ observations and $d = 5$ features, what are the dimensions of $X^T X$ versus $X X^T$?

Answer: Since $X$ is $10,000 \times 5$, the feature covariance product $X^T X$ is a tiny $5 \times 5$ matrix, whereas the Gram matrix $X X^T$ is a massive $10,000 \times 10,000$ matrix.

• Matrix Compactness: $C = \frac{1}{n}X^T X$ summarizes $n$ points in $\mathbb{R}^d$ into a symmetric $d \times d$ matrix.
• Scale Invariance of Directions: Dividing by $n$ or $n-1$ leaves all eigenvectors and explained variance ratios unchanged.
• Canonical References: See Wikipedia Reference on Sample Mean and Covariance.

Course Navigation & Chapter Links

This article is part of the complete 24-part Synapticz public course: Intuitive Eigenvalues, Eigenvectors, and Principal Component Analysis (Top-Down & Bottom-Up).

• Chapter 4 Overview: Chapter 4: What a Covariance Matrix Really Is — From Scatter Clouds to Orthogonal Stretch Axes
• Microlearning 4.1: 4.1 Constructing the Sample Covariance Matrix C = (1/n)XᵀX from Centered Data
• Microlearning 4.2: 4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction
• Microlearning 4.3: 4.3 The Real Spectral Theorem: Why Symmetric Covariance Matrices Have Orthogonal Eigenvectors
• Previous Lesson: Chapter 4: What a Covariance Matrix Really Is — From Scatter Clouds to Orthogonal Stretch Axes
• Next Lesson: 4.2 Quadratic Forms uᵀCu: Measuring Directional Variance in Any Direction