The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1040–1051 August 1–6, 2021. ©2021 Association for Computational Linguistics 1040 The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin David R. Cheriton School of Computer Science, University of Waterloo Vector Institute for Artificial Intelligence {ji.xin,r33tang,yaoliang.yu,jimmylin}@uwaterloo.ca Abstract In selective prediction, a classifier is allowed to abstain from making predictions on low- confidence examples. Though this setting is interesting and important, selective prediction has rarely been examined in natural language processing (NLP) tasks. To fill this void in the literature, we study in this paper selec- tive prediction for NLP, comparing different models and confidence estimators. We fur- ther propose a simple error regularization trick that improves confidence estimation without substantially increasing the computation bud- get. We show that recent pre-trained trans- former models simultaneously improve both model accuracy and confidence estimation ef- fectiveness. We also find that our proposed regularization improves confidence estimation and can be applied to other relevant scenarios, such as using classifier cascades for accuracy– efficiency trade-offs. Source code for this pa- per can be found at https://github.com/ castorini/transformers-selective. 1 Introduction Recent advances in deep learning models have pushed the frontier of natural language process- ing (NLP). Pre-trained language models based on the transformer architecture (Vaswani et al., 2017) have improved the state-of-the-art results on many NLP applications. Naturally, these models are de- ployed in various real-world applications. However, one may wonder whether they are always reliable, as pointed out by Guo et al. (2017) that modern neural networks, while having better accuracy, tend to be overconfident compared to simple networks from 20 years ago. In this paper, we study the problem of selective prediction (Geifman and El-Yaniv, 2017) in NLP. Under the setting of selective prediction, a model is allowed to abstain from making predictions on uncertain examples (Figure 1) and thereby reduce I enjoyed movies. Positive Selective Classifer Selective Classifer I enjoyed movies before watching this. Abstain ? Figure 1: Example of a selective classifier that makes a prediction for a confident example (left) and abstains for an uncertain one (right). the error rate. This is a practical setting in a lot of realistic scenarios, such as making entailment judgments for breaking news articles in search en- gines (Carlebach et al., 2020) and making critical predictions in medical and legal documents (Zhang et al., 2019). In these cases, it is totally acceptable, if not desirable, for the models to admit their un- certainty and call for help from humans or better (but more costly) models. Under the selective prediction setting, we con- struct a selective classifier by pairing a standard classifier with a confidence estimator. The confi- dence estimator measures how confident the model is for a certain example, and instructs the classifier to abstain on uncertain ones. Naturally, a good con- fidence estimator should have higher confidence for correctly …

Content truncated to 500 words