Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10659–10673 July 9-14, 2023 ©2023 Association for Computational Linguistics Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification Yunhua Zhou, Jianqiang Yang, Pengyu Wang, Xipeng Qiu∗ School of Computer Science, Fudan University {zhouyh20,xpqiu}@fudan.edu.cn yjqiang1@outlook.com pywang22@m.fudan.edu.cn Abstract Out-of-domain (OOD) intent classification is an active field of natural language understand- ing, which is of great practical significance for intelligent devices such as the Task-Oriented Dialogue System. It mainly contains two chal- lenges: it requires the model to know what it knows and what it does not know. This pa- per investigates “overthinking” in the open- world scenario and its impact on OOD intent classification. Inspired by this, we propose a two-birds-one-stone method, which allows the model to decide whether to make a decision on OOD classification early during inference and can ensure accuracy and accelerate inference. At the same time, to adapt to the behavior of dynamic inference, we also propose a training method based on ensemble methods. In ad- dition to bringing certain theoretical insights, we also conduct detailed experiments on three real-world intent datasets. Compared with the previous baselines, our method can not only improve inference speed, but also achieve sig- nificant performance improvements. Code is publicly available.1 1 Introduction With the increasing popularity of intelligent de- vices such as the Task-Oriented Dialogue System and the increasingly open environment they face, it becomes more and more challenging to accurately understand the intents behind the utterance of the users, on one hand, it needs to ensure the accuracy of In-domain (IND) intents, and on the other hand, it needs to be able to effectively identify OOD in- tent. These days, it has became a de facto standard to take the semantic representation of the last layer of finetuned Pre-train Models (PTMs) for intent understanding (Zhang et al., 2021b; Zhou et al., 2022). However, in the closed-world scenario (i.e., ∗Corresponding author. 1https://github.com/zyh190507/Dynamic-Ensemble-for- OOD (a) Overthinking in BERT (b) Overthinking in ALBERT Figure 1: Illustration of overthinking in the open-world scenario. The model becomes more “confident” (Pred Entropy decreases with the number of layers.), but the ability to distinguish IND and OOD decreases (the dif- ference is getting smaller). The results are obtained with the test set of CLINC-SMALL. training and test set come from the same distribu- tion), previous studies have pointed out that PTMs would be “overthinking” the semantic features of the sample (Kaya et al., 2019; Zhou et al., 2020), i.e., the representation of the sample may become too complex after reaching the last layer through multiple stacked transformer layers, thus affecting the final decision of the model. Naturally, in the open-world scenario (i.e., the condition of the same distribution is not main- tained), will the PTMs be “overthinking” the dif- ference between IND and OOD intents? We want to explore the evolution of the model’s ability to distinguish IND and OOD through the forward pro- cess of two widely used models …

Content truncated to 500 words