Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2023 - 2036 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation Chenze Shao1,2, Yang Feng1,2∗ 1 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences {shaochenze18z, fengyang}@ict.ac.cn Abstract Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called catas- trophic forgetting, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catas- trophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catas- trophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples at- tract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model up- date, so we name this problem imbalanced training. To alleviate this problem, we pro- pose Complementary Online Knowledge Dis- tillation (COKD), which uses dynamically up- dated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experi- mental results on multiple machine translation tasks show that our method successfully alle- viates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.1 1 Introduction Neural Machine Translation (NMT) has achieved impressive translation performance on many benchmark datasets in the past few years (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). In the domain adaptation task where we have large-scale out- domain data to improve the in-domain translation performance, continual learning, which is also referred to as fine-tuning, is often employed to ∗Corresponding author: Yang Feng 1Code is available at https://github.com/ictnlp/COKD. transfer the out-domain knowledge to in-domain (Luong and Manning, 2015). After fine-tuning, the model performs well in in-domain translation, but there is significant performance degradation in out-domain translation because it “forgets” the previously learned knowledge. This phenomenon is called catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999) and has attracted a lot of attention (Goodfellow et al., 2013; Kirkpatrick et al., 2017; Li and Hoiem, 2017; Lee et al., 2017). In this work, we observe that catastrophic forget- ting not only occurs in continual learning but also affects the traditional static training. To be specific, the final model pays imbalanced attention to train- ing samples. At the end of training, the recently exposed samples attract more attention and tend to have lower losses, while earlier samples are par- tially forgotten by the model and have higher losses. In short, training samples receive imbalanced atten- tion from the model, which mainly depends on the time when the model last saw the training sample (i.e., the …

Content truncated to 500 words