Math for Machine Learning - Chapter 15: Singular Value Decomposition (SVD): Advanced Concepts and Applications Quiz

This quiz rigorously assesses your understanding of Singular Value Decomposition (SVD), often hailed as the 'Swiss Army knife' of linear algebra. SVD breaks down any matrix into a sequence of geometric transformations: rotation, scaling, and another rotation. It provides a profound generalization of eigendecomposition, extending its utility to non-square matrices and offering unparalleled insights into the true geometry of linear transformations. The questions delve into the fundamental properties of SVD components ($U$, $S$, $V^T$), its intimate connections to matrix rank, column space, and null space, and its pivotal role in diverse applications such as Principal Component Analysis (PCA), recommender systems, and low-rank data approximations. Expect challenging questions that demand a deep conceptual grasp, advanced mathematical reasoning, and an ability to apply SVD principles to complex scenarios.

Key Formulae:

1. SVD of a matrix $A$: $A = U S V^T$

   • $U$: $m \times m$ orthogonal matrix (columns are left singular vectors)

   • $S$: $m \times n$ diagonal matrix with singular values ($\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_{\min(m,n)} \ge 0$) on the diagonal.

   • $V^T$: $n \times n$ orthogonal matrix (rows are transposes of right singular vectors).

2. Relationship to eigenvalues:

   • $A^T A = V S^T S V^T$

   • $A A^T = U S S^T U^T$

   • The non-zero singular values of $A$ are the square roots of the non-zero eigenvalues of $A^T A$ (or $A A^T$).

3. Best rank-$k$ approximation (Eckart-Young theorem):

   • $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$

   • $A_k$ is the best rank-$k$ approximation of $A$ in the Frobenius norm ($||A - A_k||_F$) and spectral norm ($||A - A_k||_2$).