Mitigating Uncertainty in Document Classification

Proceedings of NAACL-HLT 2019, pages 3126–3136 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3126 Mitigating Uncertainty in Document Classification Xuchao Zhang†‡, Fanglan Chen†, Chang-Tien Lu†, Naren Ramakrishnan† †Discovery Analytics Center, Virginia Tech, Falls Church, VA, USA ‡NEC Laboratories America, Inc, Princeton, NJ, USA †{xuczhang, fanglanc, ctlu, naren}@vt.edu, ‡xuczhang@nec-labs.com Abstract The uncertainty measurement of classifiers’ predictions is especially important in applica- tions such as medical diagnoses that need to ensure limited human resources can focus on the most uncertain predictions returned by ma- chine learning models. However, few existing uncertainty models attempt to improve over- all prediction accuracy where human resources are involved in the text classification task. In this paper, we propose a novel neural-network- based model that applies a new dropout- entropy method for uncertainty measurement. We also design a metric learning method on feature representations, which can boost the performance of dropout-based uncertainty methods with smaller prediction variance in accurate prediction trials. Extensive experi- ments on real-world data sets demonstrate that our method can achieve a considerable im- provement in overall prediction accuracy com- pared to existing approaches. In particular, our model improved the accuracy from 0.78 to 0.92 when 30% of the most uncertain pre- dictions were handed over to human experts in “20NewsGroup” data. 1 Introduction Machine learning algorithms are gradually taking over from the human operators in tasks such as machine translation (Bahdanau et al., 2014), opti- cal character recognition (Mithe et al., 2013), and face recognition (Parkhi et al., 2015). However, some real-world applications require higher accu- racy than the results achieved by state-of-the-art algorithms, which makes it difficult to directly ap- ply these algorithms in certain scenarios. For ex- ample, a medical diagnosis system (van der West- huizen and Lasenby, 2017) is expected to have a very high accuracy to support correct decision- making for medical practitioners. Although do- main experts can achieve a high performance in these challenging tasks, it is not always feasible to rely on limited and expensive human input for large-scale data sets. Therefore, if we have a model with 70% prediction accuracy, it is intu- itive to ask what percentage of the data should be handed to domain experts to achieve an overall ac- curacy rate above 90%? To maximize the value of limited human resources while achieving de- sirable results, modeling uncertainty accurately is extremely important to ensure that domain experts can focus on the most uncertain results returned by machine learning models. Most existing uncertainty models are based on Bayesian models, which are not only time- consuming but also unable to handle large-scale data sets. Deep Neural networks (DNNs) have attracted increasing attention in recent years and have been reported to achieve state-of-the-art per- formance in various machine learning tasks (Yang et al., 2016; Iyyer et al., 2014). However, un- like probabilistic models, DNNs are still at the early development stage in regards to providing the model uncertainty in their predictions. For those seeking to address the prediction uncertainty in …

Content truncated to 500 words