Learning with norm-based neural networks: model capacity, function spaces, and computational-statistical gaps

Abstract: In this talk, I will discuss some fundamental questions in modern machine learning, starting from “suitable” model capacities and then focusing on norm-based capacity as well as the induced Barron spaces from statistical to computational sides. First, I will talk about the statistical efficiency (w.r.t. metric entropy) of learning in Barron space to achieve the "best" trade-off between \epsilon-covering and the input dimension. Second, to track computational-statistical gap, I will briefly introduce a fundamental problem in complexity theory and learning theory: learning with multiple ReLU neurons (additive model, as a special case of Barron functions) via gradient descent. This analysis aims to shed light on how student neurons evolve into teacher neurons after weak recovery. The goal is to (partially) answer the following question: Which function class can be efficiently learned by (two-layer) neural networks trained by gradient descent?

(Joint work with Leello Dadi, Zhenyu Zhu, Volkan Cevher)

Bio: Dr. Fanghui Liu is currently an assistant professor at University of Warwick, UK, and a member of Centre for Discrete Mathematics and its Applications (DIMAP). His research interests include machine learning theory as well as theoretical-oriented applications. He was a recipient of AAAI24 New Faculty Award, DAAD AINeT Fellowship 2024, Rising Star in AI (KAUST 2023). He has presented three tutorials at ISIT’24, CVPR’23 and ICASSP’23, respectively. He is an area chair of ICLR, AISTATS, AAMAS and will organize a workshop at NeurIPS 2024 on fine-tuning. Prior to his current position, he worked as a postdoc researcher at EPFL (2021-2023) and KU Leuven (2019-2023), respectively. He received his PhD from Shanghai Jiao Tong University (China) in 2019 and bachelor’s degree from Harbin Institute of Technology (China) in 2014.