FOLDS seminar: Dynamics of large-step gradient descent near flat minima Amy Gutmann Hall, Room 414 3333 Chestnut Street, Philadelphia, United States
Zoom link: https://upenn.zoom.us/j/98220304722 Classically, convergence guarantees for gradient descent (GD) require that the step size be sufficiently small compared to the local curvature ("sharpness") of the objective. This "small-step" condition is routinely violated in the practical training of deep learning models, wherein the favoured "large-step" GD is observed to converge with intriguing oscillatory dynamics and an […]