Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3521–3532 August 1–6, 2021. ©2021 Association for Computational Linguistics 3521 Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training Li-Ming Zhan1 Haowen Liang1∗ Bo Liu1∗ Lu Fan1 Xiao-Ming Wu1† Albert Y.S. Lam2 Department of Computing, The Hong Kong Polytechnic University, Hong Kong S.A.R.1 Fano Labs, Hong Kong S.A.R.2 {lmzhan.zhan, michael.liang, doc-bo.liu}@connect.polyu.edu.hk {cslfan, csxmwu}@comp.polyu.edu.hk, albert@fano.ai Abstract Out-of-scope intent detection is of practical importance in task-oriented dialogue systems. Since the distribution of outlier utterances is arbitrary and unknown in the training stage, existing methods commonly rely on strong as- sumptions on data distribution such as mixture of Gaussians to make inference, resulting in ei- ther complex multi-step training procedures or hand-crafted rules such as confidence thresh- old selection for outlier detection. In this pa- per, we propose a simple yet effective method to train an out-of-scope intent classifier in a fully end-to-end manner by simulating the test scenario in training, which requires no assump- tion on data distribution and no additional post- processing or threshold setting. Specifically, we construct a set of pseudo outliers in the training stage, by generating synthetic outliers using inliner features via self-supervision and sampling out-of-scope sentences from easily available open-domain datasets. The pseudo outliers are used to train a discriminative clas- sifier that can be directly applied to and gen- eralize well on the test task. We evaluate our method extensively on four benchmark dialogue datasets and observe significant im- provements over state-of-the-art approaches. Our code has been released at https:// github.com/liam0949/DCLOOS. 1 Introduction Conversational system is becoming an indispens- able component in a variety of AI applications and acts as an interactive interface provided to users to improve user experience. Language understanding is essential for conversational systems to provide appropriate responses to users, and intent detection is usually the first step of language understanding. The primary goal is to identify diverse intentions ∗Equal contribution. † Corresponding author. Figure 1: t-SNE visualization of the learned embed- dings of the test samples of CLINC150. Top: Previous K-way training; Bottom: Our proposed (K + 1)-way training. Better view in color and enlarged. behind user utterances, which is often formalized as a classification task. However, intent classes defined during training are inevitably inadequate to cover all possible user intents at the test stage due to the diversity and randomness of user utterances. Hence, out-of-scope (or unknown) intent detection is essential, which aims to develop a model that can accurately identify known (seen in training) intent classes while detecting the out-of-scope classes that are not encountered during training. Due to the practical importance of out-of-scope intent detection, recent efforts have attempted to solve this problem by developing effective intent classification models. In general, previous works approach this problem by learning decision bound- aries for known intents and then using some confi- dence measure to distinguish known and unknown intents. For examples, LMCL (Lin and Xu, 2019) 3522 learns the …

Content truncated to 500 words