Calibrating Structured Output Predictors for Natural Language Processing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2078–2092 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2078 Calibrating Structured Output Predictors for Natural Language Processing Abhyuday Jagannatha1, Hong Yu1,2 1College of Information and Computer Sciences,University of Massachusetts Amherst 2Department of Computer Science,University of Massachusetts Lowell {abhyuday, hongyu}@cs.umass.edu Abstract We address the problem of calibrating predic- tion confidence for output entities of interest in natural language processing (NLP) applica- tions. It is important that NLP applications such as named entity recognition and ques- tion answering produce calibrated confidence scores for their predictions, especially if the applications are to be deployed in a safety- critical domain such as healthcare. However, the output space of such structured prediction models is often too large to adapt binary or multi-class calibration methods directly. In this study, we propose a general calibration scheme for output entities of interest in neu- ral network based structured prediction mod- els. Our proposed method can be used with any binary class calibration scheme and a neu- ral network model. Additionally, we show that our calibration method can also be used as an uncertainty-aware, entity-specific decod- ing step to improve the performance of the underlying model at no additional training cost or data requirements. We show that our method outperforms current calibration tech- niques for named-entity-recognition, part-of- speech and question answering. We also im- prove our model’s performance from our de- coding step across several tasks and bench- mark datasets. Our method improves the cal- ibration and model performance on out-of- domain test scenarios as well. 1 Introduction Several modern machine-learning based Natural Language Processing (NLP) systems can provide a confidence score with their output predictions. This score can be used as a measure of predictor confidence. A well-calibrated confidence score is a probability measure that is closely correlated with the likelihood of model output’s correctness. As a result, NLP systems with calibrated confidence can predict when their predictions are likely to be incorrect and therefore, should not be trusted. This property is necessary for the responsible deploy- ment of NLP systems in safety-critical domains such as healthcare and finance. Calibration of pre- dictors is a well-studied problem in machine learn- ing (Guo et al., 2017; Platt, 2000); however, widely used methods in this domain are often defined as binary or multi-class problems(Naeini et al., 2015; Nguyen and O’Connor, 2015). The structured out- put schemes of NLP tasks such as information ex- traction (IE) (Sang and De Meulder, 2003) and ex- tractive question answering (Rajpurkar et al., 2018) have an output space that is often too large for standard multi-class calibration schemes. Formally, we study NLP models that provide conditional probabilities pθ(y|x) for a structured output y given input x. The output can be a la- bel sequence in case of part-of-speech (POS) or named entity recognition (NER) tasks, or a span prediction in case of extractive question answer- ing (QA) tasks, or a relation prediction in case of relation extraction task. pθ(y|x) …

Content truncated to 500 words