Distinguished Lecture: Gradient optimization methods: large step-size and the edge of stability

LIVE STREAM: https://uwmadison.zoom.us/j/91797112134?pwd=ENtdvv6yzBhrtowZjjlCVqGpfpQVfu.1

Abstract: Optimization in deep learning relies on simple gradient descent algorithms. Although these methods are traditionally viewed as a time discretization of gradient flow, in practice, large step sizes---large enough to cause oscillation of the loss---exhibit performance advantages. This talk will review recent results on gradient descent with logistic loss with a step size large enough that the optimization trajectory is at the "edge of stability," and show the benefits of this initial oscillatory phase for linear functions and for two-layer networks. Based on joint work with Yuhang Cai, Michael Lindsey, Song Mei, Matus Telgarsky, Jingfeng Wu and Bin Yu.

Bio: Peter Bartlett is Professor of Statistics and Computer Science at UC Berkeley and Principal Scientist at Google DeepMind. At Berkeley, he is the Machine Learning Research Director at the Simons Institute for the Theory of Computing, Director of the Foundations of Data Science Institute, and Director of the Collaboration on the Theoretical Foundations of Deep Learning. He is President of the Association for Computational Learning, Honorary Professor of Mathematical Sciences at the Australian National University, and co-author with Martin Anthony of the book Neural Network Learning: Theoretical Foundations. He was awarded the Malcolm McIntosh Prize for Physical Scientist of the Year in Australia in 2001, was chosen as an Institute of Mathematical Statistics Medallion Lecturer in 2008, an IMS Fellow and Australian Laureate Fellow in 2011, a Fellow of the ACM in 2018, and recipient of the Chancellor's Distinguished Service Award in 2023. He was elected to the Australian Academy of Science in 2015.