A Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space

Proceedings of the 28th International Conference on Computational Linguistics, pages 1452–1460 Barcelona, Spain (Online), December 8-13, 2020 1452 A Deep Generative Distance-Based Classifier for Out-of-Domain Detection with Mahalanobis Space Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, Weiran Xu∗ Beijing University of Posts and Telecommunications, Beijing, China {xuhong,kqin,yanyuanmeng,liusihong,liuzijun,xuweiran}@bupt.edu.cn Abstract Detecting out-of-domain (OOD) input intents is critical in the task-oriented dialog system. Dif- ferent from most existing methods that rely heavily on manually labeled OOD samples, we focus on the unsupervised OOD detection scenario where there are no labeled OOD samples except for labeled in-domain data. In this paper, we propose a simple but strong generative distance- based classifier to detect OOD samples. We estimate the class-conditional distribution on feature spaces of DNNs via Gaussian discriminant analysis (GDA) to avoid over-confidence problems. And we use two distance functions, Euclidean and Mahalanobis distances, to measure the confi- dence score of whether a test sample belongs to OOD. Experiments on four benchmark datasets show that our method can consistently outperform the baselines. 1 Introduction Task-oriented dialog systems such as Google’s DialogFlow or Amazon’s Lex have become omnipresent to let people interact with machines using natural language. Detecting unknown or OOD (Out-of- Domain) intents from user queries is an essential component that aims to know when a query falls outside their range of predefined supported intents. Correctly identifying out-of-scope cases is thus crucial in deployed systems—both to avoid performing the wrong action and also to identify potential future directions for development. Different from traditional text classification tasks, the exact number of unknown intents in practical scenarios is hard to estimate by domain experts and lack of real OOD examples always leads to poor prior knowledge about these unknown intents. These characteristics make it challenging to identify OOD samples in the task-oriented dialog system. We classify the existing methods of detecting OOD intents into two main categories, supervised and unsupervised OOD detection. Supervised OOD detection (Scheirer et al., 2013; Fei and Liu, 2016; Kim and Kim, 2018; Larson et al., 2019; He et al., 2020c; Zheng et al., 2020) represents that there are extensive labeled OOD samples in the training data while unsupervised OOD detection (Breunig et al., 2000; Bendale and Boult, 2016; Hendrycks and Gimpel, 2017; Shu et al., 2017; Lee et al., 2018; Ren et al., 2019; Lin and Xu, 2019) means few or no labeled OOD samples except for labeled in-domain data. Unsupervised OOD detection makes it more complicated to identify unknown intents due to unseen and diverse semantic expressions. Besides, in the practical scenario, collecting large-scale OOD data is usually difficult and expensive compared to in-domain predefined intent data, especially when dealing with the rapidly evolving open-world environment. In this paper, we focus on the latter OOD setting, unsupervised OOD detection. For supervised OOD detection, classical methods such as (Fei and Liu, 2016; Larson et al., 2019), form a (m + 1)-class classification problem where the (m + 1)-th class represents the unseen intents. Further, Zheng et al. (2020) …

Content truncated to 500 words