On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7888–7899 July 9-14, 2023 ©2023 Association for Computational Linguistics On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing Zhengyao Gu Reed College Portland, OR Mark Hopkins Williams College Williamstown, MA Abstract We provide a survey and empirical compari- son of the state-of-the-art in neural selective classification for NLP tasks. We also provide a methodological blueprint, including a novel metric called refinement that provides a cali- brated evaluation of confidence functions for selective prediction. Finally, we supply docu- mented, open-source code to support the future development of selective prediction techniques. 1 Introduction Deep learning has brought massive improvements to natural language processing over the past decade, but neural networks still make mistakes. Accord- ingly, there is a current interest in confidence esti- mation techniques that perform well on deep neural networks. A prominent subarea of confidence esti- mation is selective prediction (El-Yaniv et al., 2010; Geifman and El-Yaniv, 2017). Selective predic- tion focuses on developing classifiers that choose to abstain when sufficiently uncertain. There is less focus on absolute measures of confidence, and more on a classifier’s ability to successfully rank its predictions, enabling techniques that maximize prediction quality given a desired yield (Geifman and El-Yaniv, 2019) or that maximize yield given a desired quality (Geifman and El-Yaniv, 2017). This paper provides a survey and rigorous em- pirical comparison of the state-of-the-art in neural selective classification (i.e. selective prediction where the underlying classifier is a neural network) specifically as it pertains to natural language pro- cessing. Our main contributions: (a) we survey a variety of recent techniques proposed in the ML and NLP literature and compare them across six classification tasks from the GLUE benchmark (Wang et al., 2018), do careful hyperparameter tun- ing for all surveyed techniques, and perform multi- ple trials of each technique to get an adequate sense of median and variance; (b) inspired by (Xin et al., 2021) and Kendall-tau distance (Kendall, 1948), we propose a simple metric called refinement that provides a calibrated measure of the performance of selective classification techniques; (c) we deter- mine that using maximum softmax probability as a confidence indicator remains a strong baseline, but Monte Carlo Dropout (Gal and Ghahramani, 2016) demonstrates significant improvement across multiple tasks and trials; (d) we release a docu- mented Python package called spred (selective prediction) to make our experiments transparent and reproducible. To facilitate evaluation of future techniques, the package provides tutorials about how to add and evaluate novel selective prediction methods. 2 Selective Prediction 2.1 Preliminaries A prediction function is a function f : X → Y that maps an instance space X to a label space Y. We refer to the output f(x) of the prediction function as its prediction for instance x ∈ X. We use the notation ✓(x) to refer to the gold prediction for a particular instance. The following denotes the correctly and incorrectly predicted instances of prediction function f …

Content truncated to 500 words