Talk: Aligning Foundation Models with the Multifaceted World

Live stream: https://uwmadison.zoom.us/j/96307085854?pwd=eRSaKBkKXLca3pK5rZEj1WOXZbI5xB.1

Abstract: Large language models (LLMs), also known as foundation models (FMs), have shown exceptional success in natural language processing (NLP), sparking significant excitement around artificial general intelligence (AGI). However, as we explore their potential to process diverse data from the multifaceted world—spanning multiple languages, varied sociocultural contexts, and non-textual modalities—these models still face significant challenges. My research aims to align these foundation models with the complex signals of the multifaceted world, focusing on three core themes: (1) aligning knowledge representations across modalities; (2) aligning model outputs with diverse human behaviors; and (3) interpreting the neural mechanisms underlying foundation models. In this talk, I will first present our latest studies on enhancing the multimodal capabilities of LLMs by aligning their knowledge representations with visual objects. Next, I will highlight our progress in understanding the cognitive capacities of LLMs and improving their alignment with human behaviors. Finally, I will envision our future directions for developing language technologies that are smarter, safer, and more inclusive across these three pivotal areas.

Bio: Prof. Junjie Hu is an Assistant Professor in the Department of Biostatistics and Medical Informatics and is affiliated with the Department of Computer Sciences at the University of Wisconsin-Madison. He earned his Ph.D. from Carnegie Mellon University's School of Computer Science, where he worked with Prof. Graham Neubig and Prof. Jaime Carbonell. His research lies at the intersection of natural language processing and machine learning, focusing on multilingual NLP, large language models, knowledge representation learning, preference alignment, and their applications in human-machine interactions. His work has been highlighted in public media outlets such as Wired and Slator.