Clustering queries by capabilities

Query clustering organizes LLM queries into groups that reflect shared latent capability demands, or in other words, the skills needed to correctly answer the query. Existing clustering methods, which primarily rely on taxonomies or embeddings, often fail to capture latent capability requirements. For example, a single “Mathematics” label can include queries requiring vastly different levels of capabilities, or even a combination of distinct capability requirements. 

We propose a new algorithm for this task that combines semantic information with a limited number of LLM “judging”. Our algorithm characterizes each cluster through a capability profile parameterized by a Bradley-Terry model (think ELO). Our clustering is ‘soft’ and can accommodate queries with mixed capability demands. It can also be used as a tool for downstream applications, such as curating fine-grained LLM rankings by capabilities or routing new unseen queries to an appropriate LLM. Our method outperforms human-labeled, embedding-based, and model-based baselines on these tasks. This is joint work with Fangzhou Wu and Richard Zhang. 

(This talk is part of the weekly Machine Learning Lunch Meetings (MLLM), held every Tuesday from 12:15 to 1:15 p.m.  Professors from Computer Sciences, Statistics, ECE, the iSchool, and other departments will discuss their latest research in machine learning, covering both theory and applications. This is a great opportunity to network with faculty and fellow researchers, learn about cutting-edge research at our university, and foster new collaborations. For the talk schedule, please visit https://sites.google.com/view/wiscmllm/home. To receive future weekly talk announcements, please subscribe to our UW Google Group at https://groups.google.com/u/1/a/g-groups.wisc.edu/g/mllm.)