Machine Learning Lunch Meeting

Everyone is invited to the weekly machine learning lunch meetings, where our faculty members from Computer Science, Statistics, ECE, and other departments will discuss their latest groundbreaking research in machine learning. This is an opportunity to network with faculty and fellow researchers and to learn about the cutting-edge research being conducted at our university.

Speaker: Fred Sala

Abstract: Perhaps the most enjoyable aspect of large pretrained models is using them zero-shot---directly asking for predictions without adapting or fine-tuning the models. Unfortunately, zero-shot prediction is often of poor-quality. Is this a fundamental limitation? In this talk, I will argue that large pretrained models often know more than they let on---and that we can help them use this information to improve prediction, without additional training or data. We do so in three ways. First, we show how to make predictions in vision-language models like CLIP more robust on-the-fly by simply asking language models for help improving them. Second, we integrate relational structures like knowledge graphs into model prediction pipelines, making up for pretraining limitations. Third, we apply these ideas to chain-of-thought prompting, showing how to automatically aggregate noisy reasoning paths. We will also discuss future possibilities for getting the most out of zero-shot models.