(Left) Phases of the gradient flow dynamics in the phase retrieval loss landscape for N going to infinity with a pictural representation of the Hessian eigenvalue distribution when varying the signal-to-noise ratio lpha. The red bar shows when an outlier exists in this distribution. (Right) Evolution of the local curvature: dynamics projected in the direction of least stability of the Hessian matrix (black arrows) in the intermediate (orange) regime of signal-to-noise ratio. Starting from an artless initial condition, gradient descent reaches a bad minimum. The green arrows indicate downward directions towards the good solution during the dynamics. At the end, the local curvature has become positive (red arrows).
Overview
How does gradient descent find good solutions in a non-convex, high-dimensional landscape? Studying phase retrieval, we connect its success to changes in local curvature, described by the Hessian. In the infinite-dimensional limit, the stability of trapping states controls whether optimization succeeds. At finite dimension, the dynamics can exploit information present at initialization to escape toward the signal before becoming trapped.
Type
Publication
arXiv:2403.02418
