(Left) Phases of the gradient flow dynamics in the phase retrieval loss landscape for N going to infinity with a pictural representation of the Hessian eigenvalue distribution when varying the signal-to-noise ratio lpha. The red bar shows when an outlier exists in this distribution. (Right) Evolution of the local curvature: dynamics projected in the direction of least stability of the Hessian matrix (black arrows) in the intermediate (orange) regime of signal-to-noise ratio. Starting from an artless initial condition, gradient descent reaches a bad minimum. The green arrows indicate downward directions towards the good solution during the dynamics. At the end, the local curvature has become positive (red arrows).
Abstract
Gradient descent is commonly used to find minima in rough landscapes, particularly in recent machine learning applications. However, a theoretical understadin of why good solutions are found remains elusive, expecially in strongly non-convex and high-dimensional settings. Here, we focus on the phrase retrieval problem as a typical example.
Type
Publication
arXiv:2403.02418
