Towards More Accurate Uncertainty Estimation In Text Classification

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8362–8372, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8362 Towards More Accurate Uncertainty Estimation In Text Classification Jianfeng He†, Xuchao Zhang†, Shuo Lei†∗, Zhiqian Chen+, Fanglan Chen†, Abdulaziz Alhamadani†, Bei Xiao‡, Chang-Tien Lu† †Discovery Analytics Center, Virginia Tech, Falls Church, VA, USA + Computer Science and Engineering, Mississippi State University, Starkville, Mississippi, USA ‡Department of Computer Science, American University, Washington, DC, USA †{jianfenghe, xuczhang, slei, fanglanc, hamdani, ctlu}@vt.edu, +zchen@cse.msstate.edu, ‡bei.xiao@american.edu Abstract The uncertainty measurement of classified re- sults is especially important in areas requir- ing limited human resources for higher accu- racy. For instance, data-driven algorithms di- agnosing diseases need accurate uncertainty score to decide whether additional but limited quantity of experts are needed for rectifica- tion. However, few uncertainty models focus on improving the performance of text classi- fication where human resources are involved. To achieve this, we aim at generating accu- rate uncertainty score by improving the confi- dence of winning scores. Thus, a model called MSD, which includes three independent com- ponents as “mix-up”, “self-ensembling”, “dis- tinctiveness score”, is proposed to improve the accuracy of uncertainty score by reducing the effect of overconfidence of winning score and considering the impact of different categories of uncertainty simultaneously. MSD can be applied with different Deep Neural Networks. Extensive experiments with ablation setting are conducted on four real-world datasets, on which, competitive results are obtained. 1 Introduction Text classification is a popular topic with broad applications. A successful and common model for text classification is Deep Neural Network (DNN). However, some real-world applications expect re- sults with higher accuracy than the ones achieved by state-of-the-art algorithms. Hence, the most uncertain predictions need domain experts for fur- ther decisions (Zhang et al., 2019). To efficiently leverage the limited human resources, it is essential to calculate uncertainty score of the model predic- tion, which quantifies how unconfident the model prediction is. This paper aims at generating more accurate uncertainty score through DNNs in the text classification with human involvement in the ∗Corresponding author. testing process. This is different from active learn- ing, which involves experts in the training process. Though various metrics of the uncertainty score have been studied (Dong et al., 2018; Wang et al., 2019; Shen et al., 2019; Xiao and Wang, 2019; Ku- mar et al., 2019), the existing metrics directly or indirectly depend on winning score, which is the maximum probability in a semantic vector (softmax vector from the last layer of a DNN model) (Thu- lasidasan et al., 2019). Therefore, improving Con- fidence of Winning Score (CWS), which describes how confident the winning score matches the sam- ple uncertainty and represents the accuracy of the winning score, is helpful to improve the accuracy of uncertainty score. To show the effect of im- proving CWS, this paper considers a basic way to measure uncertainty score, which is the reciprocal of winning score (Snoek et al., 2019). However, we face two challenges in improving CWS: …

Content truncated to 500 words