Understanding Deep Models with Teacher-student Setting

While deep networks show impressive empirical performance, a lot of mysteries remain on how it learns from supervised data and unlabeled data purely with self-supervision. As a result, deep models are often treated as a black-box mapping, cranking out results that might or might not be correct on novel inputs. In this talk, I will give an overview about the teacher-student setting, a methodology used to study how deep models work. In supervised learning, the teacher-student setting treats the labels as generated by a hidden teacher network which typically has a similar architecture as the student network. The existence of the teacher network gives a good reference of the student learning, facilitating the understanding of many of its properties. Besides the overview, I will also introduce some of our very recent works that apply this setting to supervised and self-supervised learning and show interesting insights on how the deep models work.
 

Yuandong Tian is a Research Scientist and Manager in Facebook AI Research, working on deep reinforcement learning, multi-agent learning and its applications, and theoretical analysis of deep models. He is the lead scientist and engineer for ELF OpenGo and DarkForest Go projects. Prior to that, he was a researcher and engineer in Google Self-driving Car team in 2013-2014. He received a Ph.D in Robotics Institute, Carnegie Mellon University in 2013, Bachelor and Master degree of Computer Science in Shanghai Jiao Tong University. He is the recipient of 2013 ICCV Marr Prize Honorable Mentions.

Host: Sharon Li