MML-16-02: Activation Functions & Derivative Singularities – Practice Quiz

Analyze the calculus, geometry, and optimization diagnostics of neural network activation functions (Sigmoid, Tanh, ReLU, Leaky ReLU, PReLU, ELU, Softplus, Swish/SiLU, and GELU): exact analytical derivatives, second derivatives, saturation bounds, vanishing gradients, non-zero-centered gradient zig-zagging, subdifferentials at derivative singularities, Dying ReLU mechanics, and non-monotonic smooth gating in modern Transformers. Aligns with Prerequisites: QS 16.1; Cognitive Level: Tier 2–3.