Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2669–2678 June 6–11, 2021. ©2021 Association for Computational Linguistics 2669 Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding Ting Hua Yilin Shen Changsheng Zhao Samsung Research America {ting.hua,yilin.shen,changsheng.z,yenchang.hsu,hongxia.jin}@samsung.com Yen-Chang Hsu Hongxia Jin Abstract Domain classification is the fundamental task in natural language understanding (NLU), which often requires fast accommodation to new emerging domains. This constraint makes it impossible to retrain all previous domains, even if they are accessible to the new model. Most existing continual learning approaches suffer from low accuracy and performance fluctuation, especially when the distributions of old and new data are significantly differ- ent. In fact, the key real-world problem is not the absence of old data, but the ineffi- ciency to retrain the model with the whole old dataset. Is it potential to utilize some old data to yield high accuracy and maintain stable per- formance, while at the same time, without in- troducing extra hyperparameters? In this paper, we proposed a hyperparameter-free continual learning model for text data that can stably produce high performance under various en- vironments. Specifically, we utilize Fisher in- formation to select exemplars that can “record” key information of the original model. Also, a novel scheme called dynamical weight consol- idation is proposed to enable hyperparameter- free learning during the retrain process. Ex- tensive experiments demonstrate that baselines suffer from fluctuated performance and there- fore useless in practice. On the contrary, our proposed model CCFI significantly and con- sistently outperforms the best state-of-the-art method by up to 20% in average accuracy, and each component of CCFI contributes ef- fectively to overall performance. 1 Introduction Catastrophic forgetting is the well-known Achilles’ heel of deep neural networks, that the knowledge learned from previous tasks will be forgotten when the networks are retrained to adapt to new tasks. Al- though this phenomenon has been noticed as early as the birth of neural networks (French, 1999; Mc- Closkey and Cohen, 1989), it didn’t attract much attention until deep neural networks have achieved impressive performance gains in various applica- tions (LeCun et al., 2015; Krizhevsky et al., 2012). Domain classification is the task that mapping the spoken utterances to natural language under- standing domains. It is widely used in intelligent personal digital assistant (IPDA) devices, such as Amazon Alexa, Google Assistant, and Microsoft Cortana. As many IPDAs now allow third-party developers to build and integrate new domains (Ku- mar et al., 2017), these devices are eager to con- tinual learning technologies that can achieve high performance stably (Kim et al., 2018a,b). How- ever, most traditional IPADs only work with well- separated domains built by specialists (Tur and De Mori, 2011) or customized designed for spe- cific datasets (Li et al., 2019). There is still a lack of continual learning methods that capable of general domain classification. Most previous approaches capable of continual learning focus on the scenario that the new model should be retrained …

Content truncated to 500 words