Consistent Estimators for Learning to Defer to an Expert

Abstract

Learning algorithms are often used in conjunction with expert decision makers in practical scenarios, however this fact is largely ignored when designing these algorithms. In this paper we explore how to learn predictors that can either predict or choose to defer the decision to a downstream expert. Given only samples of the expert’s decisions, we give a procedure based on learning a classifier and a rejector and analyze it theoretically. Our approach is based on a novel reduction to cost sensitive learning where we give a consistent surrogate loss for cost sensitive learning that generalizes the cross entropy loss. We show the effectiveness of our approach on a variety of experimental tasks.

Publication
Proceedings of the Thirty-Seventh International Conference on Machine Learning (ICML)
Hussein Mozannar
Hussein Mozannar
PhD Student

Hussein’s interests focus on human-centric aspects of machine learning, namely how to integrate expert decision makers into machine learning pipelines while ensuring fairness and an understanding of long-term consequences.

David Sontag
David Sontag
Associate Professor of EECS

My research focuses on advancing machine learning and artificial intelligence, and using these to transform health care.

Related