Talk: Beyond UDFs: Optimizing LLM-Powered Operators in Data Systems

Zoom: https://uwmadison.zoom.us/j/96432556503?pwd=q7C8WD2T8zUYM2Ap9DGDOkUVXqI1jq.1

Abstract: Database vendors are racing to add LLM-powered relational operators, such as AI.FILTER, AI.JOIN, and AI.CLASSIFY, so users can finally query their unstructured data. Most implement these operators as black-box UDFs that call an external API, at great expense. In this talk, we open the black box. First, I will discuss query optimization techniques from DocETL, our system for processing unstructured data with LLMs, and its follow-up work. These include new rewrite rules and search algorithms that account for the idiosyncrasies of LLMs. Then, I will discuss our ongoing work on Quail, an execution engine for AI-SQL that runs LLM inference on the GPU itself, optimizing forward passes and memory management around the query plan. DocETL has over 4.1k GitHub stars and is used by many companies, as well as public defenders in multiple California counties. Quail was released a few weeks ago and has far fewer GitHub stars, but we’re excited about where it’s headed.

Bio: Shreya Shankar is a CS researcher broadly interested in databases and human-AI interaction. In 2027, she will join CMU as an Assistant Professor in the CS department (and HCII, by courtesy), affiliated with the CMU database group. She received her PhD in EECS from UC Berkeley, where she built DocETL, an open source system for processing unstructured data, and received several fancy paper awards (Best of SIGMOD 2026, CHI 2026 Best Paper, UIST 2025 Best Paper Honorable Mention) and fellowships.