Energy-based Unknown Intent Detection with Data Manipulation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2852–2861 August 1–6, 2021. ©2021 Association for Computational Linguistics 2852 Energy-based Unknown Intent Detection with Data Manipulation Yawen Ouyang, Jiasheng Ye, Yu Chen, Xinyu Dai ∗, Shujian Huang and Jiajun Chen National Key Laboratory for Novel Software Technology, Nanjing University, China {ouyangyw,yejiasheng,cheny98}@smail.nju.edu.cn {daixinyu,huangsj,chenjj}@nju.edu.cn Abstract Unknown intent detection aims to identify the out-of-distribution (OOD) utterance whose in- tent has never appeared in the training set. In this paper, we propose using energy scores for this task as the energy score is theoretically aligned with the density of the input and can be derived from any classifier. However, high- quality OOD utterances are required during the training stage in order to shape the energy gap between OOD and in-distribution (IND), and these utterances are difficult to collect in practice. To tackle this problem, we propose a data manipulation framework to Generate high-quality OOD utterances with importance weighTs (GOT). Experimental results show that the energy-based detector fine-tuned by GOT can achieve state-of-the-art results on two benchmark datasets. 1 Introduction Unknown intent detection is a realistic and chal- lenging task for dialogue systems. Detecting out- of-distribution (OOD) utterances is critical when employing dialogue systems in an open environ- ment. It can help dialogue systems gain a better understanding of what they do not know, which prevents them from yielding unrelated responses and improves user experience. A simple approach for this task relies on the soft- max confidence score and achieves promising re- sults (Hendrycks and Gimpel, 2017). The softmax- based detector will classify the input as OOD if its softmax confidence score is smaller than the threshold. Nevertheless, further works demonstrate that using the softmax confidence score might be problematic as the score for OOD inputs can be arbitrarily high (Louizos and Welling, 2017; Lee et al., 2018). ∗ Corresponding author. Utterance: How much did I spend this week Intent: spending history Locating Module Intent-related Word(s): How much did I spend this week Generating Module OOD Utterance 1: How much did I drink this week OOD Utterance 2: How much did I lose this week …… Weighting Module Figure 1: An overview of our framework GOT. For the utterance “How much did I spend this week” from CLINC150 dataset (Larson et al., 2019). Our locating module locates the intent-related word “spend”. And then our generating module generates words “drink”, “lose” to replace it and obtains OOD utterances. Fi- nally, our weighting module assigns a weight for each of OOD utterances. Another appealing approach is to use genera- tive models to approximate the distribution of in- distribution (IND) training data and use the likeli- hood score to detect OOD inputs. However, Ren et al. (2019) and Gangal et al. (2019) find that like- lihood scores derived from such models are prob- lematic for this task as they can be confounded by background components in the inputs. In this paper, we propose using energy scores (Liu et al., 2020) for unknown intent detection. The benefit is …

Content truncated to 500 words