Zoom link: https://upenn.zoom.us/j/98220304722

Classically, convergence guarantees for gradient descent (GD) require that the step size be sufficiently small compared to the local curvature (“sharpness”) of the objective. This “small-step” condition is routinely violated in the practical training of deep learning models, wherein the favoured  “large-step” GD is  observed to converge with intriguing oscillatory dynamics and an implicit bias toward solutions of low curvature known as “flat minima”. In this talk, I will present a new geometric perspective on these phenomena. In particular, I will indicate how the loss geometry of certain multilayer models enables the proof of a new normal form for large-step GD near the global minima manifold. In this normal form, large-step GD is revealed to act as Riemannian GD on the objective sharpness along the solution manifold coupled to a flip bifurcation in the directions orthogonal to the solution manifold, enabling new local convergence theorems for large-step GD near flat minima.