Causally Interpreting Vision–Language Models: From Semantic Edits to Neuron Control

Recent advances in large vision–language models (VLMs) such as LLaVA have demonstrated impressive multimodal capabilities, yet their internal decision processes remain largely opaque. In this talk, I present some initial steps toward making these systems more transparent and controllable through a unified causal interpretability framework. We begin with semantic-level visual interventions and attention-head analysis, and then extend the investigation to feed-forward network neuron attribution and steering. By connecting these levels, our empirical studies provide a coherent causal perspective on how VLMs integrate visual and textual information—translating mechanistic understanding into practical tools for model diagnosis and performance improvement.

(This talk is part of the weekly Machine Learning Lunch Meetings (MLLM), held every Tuesday from 12:15 to 1:15 p.m.  Professors from Computer Sciences, Statistics, ECE, the iSchool, and other departments will discuss their latest research in machine learning, covering both theory and applications. This is a great opportunity to network with faculty and fellow researchers, learn about cutting-edge research at our university, and foster new collaborations. For the talk schedule, please visit https://sites.google.com/view/wiscmllm/home. To receive future weekly talk announcements, please subscribe to our UW Google Group at https://groups.google.com/u/1/a/g-groups.wisc.edu/g/mllm.)