Continual Learning for Neural Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3964–3974 June 6–11, 2021. ©2021 Association for Computational Linguistics 3964 Continual Learning for Neural Machine Translation Yue Cao,1,2,3 Hao-Ran Wei,4 Boxing Chen,4 Xiaojun Wan1,2,3 1Wangxuan Institute of Computer Technology, Peking University 2Center for Data Science, Peking University 3The MOE Key Laboratory of Computational Linguistics, Peking University 4Alibaba Group {yuecao,wanxiaojun}@pku.edu.cn, {funan.whr,boxing.cbx}@alibaba-inc.com Abstract Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT mod- els are usually trained on a general domain cor- pus and then fine-tuned by continuing training on the in-domain corpus. However, this bears the risk of catastrophic forgetting that the per- formance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT mod- els. We consider a scenario where the train- ing is comprised of multiple stages and pro- pose a dynamic knowledge distillation tech- nique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and pro- pose a bias-correction module to eliminate the bias. We conduct experiments on three repre- sentative settings of NMT application. Experi- mental results show that the proposed method achieves superior performance compared to baseline models in all settings.1 1 Introduction Continual learning, which is also referred to as incremental learning or lifelong learning, is a learn- ing paradigm that allows the agent to continuously learn from new knowledge without forgetting pre- viously learned knowledge. Humans naturally have the ability to continually acquire knowledge while preserving old knowledge throughout their lifes- pan. In real-world applications, data is usually given in a continuous stream form, and only part of the data is available at the beginning of training. There- fore, the ability to learn from continuous streams of information is crucial for artificial intelligence systems. However, continual learning remains a 1This work was done when Yue Cao was an intern at Al- ibaba. Codes are available at https://github.com/caoy1996/CL- NMT. big challenge for artificial intelligence systems and models since they suffer from the problem of catas- trophic forgetting (French, 1993), i.e., the learn- ing of new tasks may cause the model to forget the knowledge learned from previous tasks. This phenomenon typically leads to a significant perfor- mance decrease in previously learned tasks. One trivial solution to avoid catastrophic forgetting is to retrain from scratch by combining old and new tasks. However, this methodology is computation- ally inefficient and needs to store old data all the time. Recently, continual learning has received in- creasing attention in the artificial intelligence filed. Most of existing works focus on computer vision tasks (Zenke et al., 2017; Aljundi et al., 2017; Triki et al., 2017; Hou et al., 2018; Aljundi et al., 2018; Hou et al., 2019; Wu et al., 2019). In the natural language processing area, several methods have been proposed to alleviate the prob- lem …

Content truncated to 500 words