Statistics Seminar

Abstract: Machine learning (ML) is increasingly being used to support decision-making in many organizational settings. However, there is currently a gap between the design and evaluation of ML algorithms and the functional role of these algorithms as tools for decision support. The first part of the talk will highlight the role of humans-in-the-loop, and the importance of evaluating decisions instead of predictions, through a study of the adoption of a risk assessment tool in child maltreatment hotline screenings. The second part of the talk will focus on the gap between the construct of interest and the proxy that the algorithm optimizes for. Using a proposed machine learning methodology that extracts knowledge from experts’ historical decisions, we show that in the context of child maltreatment hotline screenings (1) there are high-risk cases whose risk is considered by the experts but not wholly captured in the target labels used to train a deployed model, and (2) we can bridge this gap if we purposefully design with this goal in mind.