Math for Machine Learning - Chapter 13: Orthogonality and Orthogonal Matrices

This quiz assesses deep understanding of orthogonality and orthogonal matrices, crucial concepts in the mathematical foundations of machine learning and numerical analysis. Questions cover theoretical properties, computational implications, and applications in algorithms like QR decomposition and attention mechanisms. Learners will encounter challenging problems requiring a strong grasp of linear algebra, numerical stability, and their relevance in advanced machine learning contexts. Key formulas include:

• Orthogonal Matrix Definition: $$Q^T Q = QQ^T = I$$

• Preservation of Length (Euclidean Norm): $$\|Qx\| = \|x\|$$

• Preservation of Inner Product/Angle: $$ (Qx) \cdot (Qy) = x \cdot y $$

• Determinant of Orthogonal Matrix: $$\det(Q) = \pm 1$$

• Orthonormal Basis Property: $$u_i^T u_j = \delta_{ij}$$ (Kronecker delta)

• Projection onto Subspace $$W$$ with orthonormal basis $$U$$: $$P_W = UU^T$$

• QR Decomposition: $$A = QR$$, where $$Q$$ is orthogonal/has orthonormal columns and $$R$$ is upper triangular.

• Least Squares Solution via QR: For $$Ax \approx b$$, the solution $$\hat{x}$$ minimizes $$\|Ax - b\|^2$$ and satisfies $$R\hat{x} = Q^T b$$.

• Scaled Dot-Product Attention: $$Attention(Q, K, V) = softmax(\frac{QK^T}{\sqrt{d_k}})V$$