Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4315–4326 Barcelona, Spain (Online), December 8-13, 2020 4315 Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation Shuhao Gu1,2, Yang Feng1,2∗ 1 Key Laboratory of Intelligent Information Processing, Instiute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences {gushuhao19b,fengyang}@ict.ac.cn Abstract Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data which may have a different distribution, e.g. a different domain. Although many methods have been proposed to solve this problem, we cannot get to know what causes this phenomenon yet. Under the background of domain adaptation, we investigate the cause of catastrophic forgetting from the perspectives of modules and parameters (neurons). The investigation on the modules of the NMT model shows that some modules have tight relation with the general-domain knowledge while some other modules are more essential in the domain adaptation. And the investigation on the parameters shows that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general-domain. We conduct experiments across different language pairs and domains to ensure the validity and reliability of our findings. 1 Introduction Neural machine translation (NMT) models (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) have achieved state-of-the-art results and have been widely used in many fields. Due to numerous parameters, NMT models can only play to their advantages based on large-scale training data. However, in practical ap- plications, NMT models often need to perform translation for some specific domain with only a small quantity of in-domain data available. In this situation, continual training (Luong and Manning, 2015), which is also referred to as fine-tuning, is often employed to improve the in-domain translation per- formance. In this method, the model is first trained with large-scale general-domain training data and then continually trained with the in-domain data. With this method, the in-domain performance can be improved greatly, but unfortunately, the general-domain performance decline significantly, since NMT models tend to overfit to frequent observations (e.g. words, word co-occurrences, translation patterns) in the in-domain data but forget previously learned knowledge. This phenomenon is called catastrophic forgetting. Figure 1 shows the performance trends on the in-domain and general-domain. Many methods have been proposed to address the catastrophic forgetting problem under the scheme of fine-tuning. Freitag and Al-Onaizan (2016) ensembles the general-domain model and the fine-tuned model together so that the integrated model can consider both domains. Dakwale and Monz (2017) in- troduces domain-specific output layers for both of the domains and thus the domain-specific features of the two domains can be well preserved. Thompson et al. (2019), Barone et al. (2017), and Khayrallah et al. (2018) propose regularization-based methods that introduce an additional loss to …

Content truncated to 500 words