MadS&P Weekly Seminar - Guest Speaker: Zhuolin Yang

Abstract: Equipped with smartphones, tablets, and virtual reality headsets, we can now perform a wide variety of tasks from virtually anywhere. However, this newfound freedom and flexibility come with its own challenges, particularly in terms of personal privacy. One emerging threat is sophisticated machine learning (ML) attacks that break privacy assumptions by extracting previously undetectable patterns carried by our daily actions. 

In this talk, I will discuss two of our recent projects on identifying and mitigating real-world privacy threats. First, we identify a general keystroke inference attack in the physical world, where the attacker does not rely on any prior knowledge or labeled data on the target user or their device.  Just by analyzing a 10-min video of the target’s typing hands from meters away, the attacker can extract over 90% of their typed content. This is done by applying a self-supervised learning pipeline to the video, which curates labeled data from the video and train DNN models to achieve robust inference results. We then extend this attack into virtual reality. By screen-recording the target’s avatar for 10 min, the attacker can also recover the typed content. This is made possible by the addition of transformer models designed to mitigate digital noise in avatar motion capture.  Finally, I will also discuss ongoing projects that explore new forms of privacy attacks in VR systems.

Bio:  Zhuolin is a 5th year CS Ph.D. student at SAND Lab, University of Chicago, co-advised by Prof. Ben Y. Zhao and Prof. Heather Zheng. Zhuolin's research focuses on both exploring how human motion can disclose private/sensitive information during interactions with everyday computing devices, and designing interactive systems that leverage human motions as a means to safeguard user privacy. For more see: 

https://zhuoliny.github.io/