Math for Machine Learning - Chapter 11: Eigenvalues and Eigenvectors

This quiz is designed to test a deep understanding of eigenvalues and eigenvectors, focusing on their theoretical foundations, geometric intuition, and properties of special matrix types relevant to machine learning. It covers advanced concepts such as algebraic and geometric multiplicity, diagonalizability, Cayley-Hamilton Theorem, Perron-Frobenius Theorem, and applications to covariance matrices, aiming to solidify expertise in 'eigen-everything'. The questions are crafted to be challenging, requiring analytical reasoning beyond simple formula recall.

Key Formulas and Concepts:

• Eigenvalue Equation: $Av = \lambda v$

• Characteristic Equation: $\det(A - \lambda I) = 0$

• Eigenspace: $E_{\lambda} = \text{Null}(A - \lambda I)$

• Algebraic Multiplicity (AM): The multiplicity of $\lambda$ as a root of the characteristic polynomial.

• Geometric Multiplicity (GM): $\dim(E_{\lambda})$. Always $1 \le GM \le AM$.

• Diagonalization: A matrix $A$ is diagonalizable if and only if $A = PDP^{-1}$, where $D$ is a diagonal matrix of eigenvalues and $P$ is an invertible matrix of eigenvectors. This occurs if and only if $GM = AM$ for all eigenvalues.

• Trace and Determinant: $\text{tr}(A) = \sum \lambda_i$, $\det(A) = \prod \lambda_i$.

• Symmetric Matrices: Real eigenvalues, orthogonal eigenvectors (for distinct eigenvalues), diagonalizable by orthogonal matrix ($A = QDQ^T$).

• Positive Definite Matrices: Symmetric, with all eigenvalues strictly positive; $x^T A x > 0$ for all non-zero $x$.

• Projection Matrices: Eigenvalues are only 0 or 1.

• Orthogonal Matrices: Eigenvalues have magnitude 1.

• Cayley-Hamilton Theorem: Every square matrix satisfies its own characteristic equation, i.e., if $p(\lambda)$ is the characteristic polynomial of $A$, then $p(A) = 0$.

• Perron-Frobenius Theorem (for positive matrices): Guarantees a unique largest positive eigenvalue (Perron root) with a strictly positive eigenvector.