Mitigating the Diminishing Effect of Elastic Weight Consolidation

Proceedings of the 29th International Conference on Computational Linguistics, pages 4568–4574 October 12–17, 2022. 4568 Mitigating the Diminishing Effect of Elastic Weight Consolidation Canasai Kruengkrai Junichi Yamagishi National Institute of Informatics, Japan {canasai,jyamagishi}@nii.ac.jp Abstract Elastic weight consolidation (EWC, Kirk- patrick et al. 2017) is a promising approach to addressing catastrophic forgetting in sequen- tial training. We find that the effect of EWC can diminish when fine-tuning large-scale pre- trained language models on different datasets. We present two simple objective functions to mitigate this problem by rescaling the com- ponents of EWC. Experiments on natural lan- guage inference and fact-checking tasks indi- cate that our methods require much smaller values for the trade-off parameters to achieve results comparable to EWC.1 1 Introduction New training data may arrive after we have spent considerable time training our model on the data at hand. A simple method for exploiting both new and old training data is to mix them and retrain the model from scratch. However, this mix-and-retrain method is neither always practical nor economi- cal, especially in academic environments where computational resources are limited. Sequential training is a potential alternative ap- proach but faces a difficult challenge called catas- trophic forgetting in which the performance on old data drastically drops when we train a model on new data. There exists a line of work that has ad- dressed this challenge (Rusu et al., 2016; Li and Hoiem, 2018; Kirkpatrick et al., 2017; Mallya et al., 2018; He and Jaeger, 2018; Zhang et al., 2020). In this paper, we are particularly interested in elas- tic weight consolidation (EWC, Kirkpatrick et al. 2017), which has been shown to be helpful for do- main adaptation (Saunders et al., 2019; Thompson et al., 2019). EWC adds a regularization term to the objec- tive function to ensure that the model works well on both new and old data. We empirically find 1Our code is available at https://github.com/ nii-yamagishilab/ewc. 101 103 105 107 109 72 75 78 80 82 85 88 90 Accuracy (%) MNLI FEVER Micro-avg Figure 1: Accuracy vs. trade-off parameter λ. We se- quentially fine-tune BERT (Bidirectional Encoder Rep- resentations from Transformers, Devlin et al. 2019) on MNLI (Williams et al., 2018) and FEVER (Thorne et al., 2018) and evaluate performance on the balanced dev sets. EWC starts to increase the accuracy of the prior dataset (MNLI) when increasing λ to 105 and yields the highest average accuracy at 107. that EWC requires unexpectedly large values for the trade-off parameter (λ) between the regular- izer and the loss to be effective when applying to pre-trained language models. Figure 1 shows such a phenomenon in which EWC has no effect in preventing catastrophic forgetting of the prior dataset (MNLI) with λ in the range of [100, 104]. We have to scale λ up to [105, 107], which is an un- usual range of hyperparameters. To the best of our knowledge, this phenomenon has not been reported in the literature. We propose two simple objective functions for mitigating …

Content truncated to 500 words