A Progressive Model to Enable Continual Learning for Semantic Slot Filling

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1279–1284, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1279 A Progressive Model to Enable Continual Learning for Semantic Slot Filling Yilin Shen1 Xiangyu Zeng2∗ Hongxia Jin1 1 Samsung Research America, Mountain View, CA, USA 2 Columbia University, New York, NY, USA 1 {yilin.shen,hongxia.jin}@samsung.com 2 xz2571@columbia.edu Abstract Semantic slot filling is one of the major tasks in spoken language understanding (SLU). After a slot filling model is trained on pre- collected data, it is crucial to continually improve the model after deployment to learn users’ new expressions. As the data amount grows, it becomes infeasible to either store such huge data and repeatedly retrain the model on all data or fine tune the model only on new data without forgetting old expressions. In this paper, we introduce a novel progressive slot filling model, ProgModel. ProgModel consists of a novel context gate that transfers previously learned knowledge to a small size expanded component; and meanwhile enables this new component to be fast trained to learn from new data. As such, ProgModel learns the new knowledge by only using new data at each time and meanwhile preserves the previously learned expressions. Our experiments show that ProgModel needs much less training time and smaller model size to outperform various model fine tuning competitors by up to 4.24% and 3.03% on two benchmark datasets. 1 Introduction Spoken language understanding (SLU) systems play a vital role in ubiquitous artificially intelligent voice-enabled personal assistants. As one of the major tasks in SLU, semantic slot filling is treated as a sequential labeling problem to map a natural language sequence x to a slot label sequence y of the same length in IOB format (Yao et al., 2014). Typically, a slot filling model is trained offline on large scale corpora with pre-collected utterances. However, such corpora usually cannot cover all possible varieties of utterances exhaustively (e.g., personalized expressions, new vocabulary, ∗ This work was done when Xiangyu Zeng was in Samsung Research. utterances for new intent, etc.) from diverse users. Thus, it is critically desirable to develop a slot filling approach with the capability of continual learning after a personal assistant is deployed. Unfortunately, existing approaches target on offline model training using a large scale training data. They are designed to either train a slot filling model independently (Yao et al., 2014; Peng et al., 2015; Kurata et al., 2016; Hakkani-T¨ur et al., 2016; Liu and Lane, 2016; Deng et al., 2019; Ray et al., 2019) or jointly with the other intent detection task in SLU (Guo et al., 2014; Liu and Lane, 2016; Zhang and Wang, 2016; Wang et al., 2018; Goo et al., 2018). Recently, (Shen et al., 2018a, 2019) developed cold start algorithms to generate training data with the hope of covering more varieties before deployment. On the other hand, (Ray et al., 2018; Shen et al., 2018b) attempt …

Content truncated to 500 words