Mitigating hallucinations in diffusion models through dynamic guidance

Diffusion models produce stunningly realistic content across modalities like images, video, and text. Yet, this power is often undermined by a critical flaw: hallucinations, where models generate samples with structural inconsistencies lying outside the true data distribution. This talk confronts this challenge by exploring two fundamental questions: Are such inconsistencies always detrimental, and when they are, how can we effectively mitigate them? Arguing that the answers are deeply intertwined, I will introduce a simple idea that leverages class labels to actively guide diffusion models away from these hallucinatory outputs. The presentation will conclude by exploring the application to more complex scenaria where class labels are uninformative or entirely absent, paving the way for more robust and reliable generative models.

(This talk is part of the weekly Machine Learning Lunch Meetings (MLLM), held every Tuesday from 12:15 to 1:15 p.m.  Professors from Computer Sciences, Statistics, ECE, the iSchool, and other departments will discuss their latest research in machine learning, covering both theory and applications. This is a great opportunity to network with faculty and fellow researchers, learn about cutting-edge research at our university, and foster new collaborations. For the talk schedule, please visit https://sites.google.com/view/wiscmllm/home. To receive future weekly talk announcements, please subscribe to our UW Google Group at https://groups.google.com/u/1/a/g-groups.wisc.edu/g/mllm.)