Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2443–2453 August 1–6, 2021. ©2021 Association for Computational Linguistics 2443 Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU Yilin Shen, Yen-Chang Hsu, Avik Ray, Hongxia Jin Samsung Research America {yilin.shen,yenchang.hsu,avik.r,hongxia.jin}@samsung.com Abstract Intent classification is a major task in spoken language understanding (SLU). Since most models are built with pre-collected in-domain (IND) training utterances, their ability to de- tect unsupported out-of-domain (OOD) utter- ances has a critical effect in practical use. Re- cent works have shown that using extra data and labels can improve the OOD detection per- formance, yet it could be costly to collect such data. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection. Our method designs a novel domain-regularized module (DRM) to reduce the overconfident phenomenon of a vanilla classifier, achieving a better generalization in both cases. Besides, DRM can be used as a drop-in replacement for the last layer in any neural network-based intent classifier, providing a low-cost strategy for a significant improvement. The evalua- tion on four datasets shows that our method built on BERT and RoBERTa models achieves state-of-the-art performance against existing approaches and the strong baselines we cre- ated for the comparisons. 1 Introduction Spoken language understanding (SLU) systems play a crucial role in ubiquitous artificially intelli- gent voice-enabled personal assistants (PA). SLU needs to process a wide variety of user utterances and carry out user’s intents, a.k.a. intent classifica- tion. Many deep neural network-based SLU mod- els have recently been proposed and have demon- strated significant progress (Guo et al., 2014; Liu and Lane, 2016; Zhang and Wang, 2016; Wang et al., 2018; Goo et al., 2018; Chen et al., 2019) in classification accuracy. These models usually apply the closed-world assumption, in which the SLU model is trained with predefined domains, and the model expects to see the same data distribution I don’t like Thriller in playlist Playlist deleted I am too cold Oven turned on Figure 1: Failure Examples of Unsupported Skills in AI Voice Assistants. The user’s utterances are out of the designed domains of the assistant. during both training and testing. However, such an assumption is not held in the practical use case of PA systems, where the system is used under a dynamic and open environment with personal ex- pressions, new vocabulary, and unknown intents that are out of the design scope. To address the challenges in open-world settings, previous works adopt varied strategies. Shen et al. (2018a, 2019c) use a cold-start algorithm to gener- ate additional training data to cover a larger variety of utterances. This strategy relies on the software developers to pre-build all possible skills. Shen et al. (2019b,a) introduce a SkillBot that allows users to build up their own skills. Recently, Ray et al. (2018, 2019); Shen et al. (2018b, 2019d) en- ables an SLU model to …

Content truncated to 500 words