Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP

Does an LSTM forget more than a CNN? An empirical study of catastrophic forgetting in NLP Gaurav Arora Afshin Rahimi School of Computing and Information Systems The University of Melbourne gaurava@student.unimelb.edu.au {arahimi,tbaldwin}@unimelb.edu.au Timothy Baldwin Abstract Catastrophic forgetting — whereby a model trained on one task is fine-tuned on a second, and in doing so, suffers a “catastrophic” drop in performance over the first task — is a hur- dle in the development of better transfer learn- ing techniques. Despite impressive progress in reducing catastrophic forgetting, we have limited understanding of how different archi- tectures and hyper-parameters affect forgetting in a network. In this paper, we aim to un- derstand factors which cause forgetting dur- ing sequential training. Our primary finding is that CNNs forget less than LSTMs. We show that max-pooling is the underlying op- eration which helps CNNs alleviate forgetting compared to LSTMs. We also found that cur- riculum learning (Bengio et al., 2009), placing a hard task towards the end of task sequence, reduces forgetting. We analysed the effect of fine-tuning contextual embeddings on catas- trophic forgetting, and found that using fixed word embeddings is preferable to fine-tuning.1 1 Introduction Transfer learning — transferring knowledge from a source task to a target task — has become an essential technique in both computer vision and NLP. Earlier attempts at transfer learning were limited in their applicability (Mou et al., 2016), as the transfer only worked for very similar tasks: if the source and target tasks were not very similar, training on the target task resulted in catastrophic forgetting (Ratcliff, 1990; McCloskey and Cohen, 1989), whereby the neural network abruptly for- gets previously-acquired knowledge during train- ing on a new task, limiting inductive transfer. ULMFit (Howard and Ruder, 2018) developed specialised techniques to reduce forgetting dur- ing the fine-tuning process, resulting in successful 1All code associated with this paper is available at https://github.com/gauravaror/catastrophic forgetting transfer learning. A general finding of this work was that uncovering underlying causes of catas- trophic forgetting can result in improved architec- tures for transfer learning. Previous studies found that using dropout (Goodfellow et al., 2014) and sharp activation functions (French, 1991) help reduce catastrophic forgetting. Sharp activation functions effectively distribute each task to different parts of the net- work. It is unknown if increasing the capacity of the network will also have a similar effect. An- other study found that using a max operation (Sri- vastava et al., 2013) in the network reduces forget- ting. There hasn’t been a study comparing forget- ting for different architectures to test if networks with the max operation have less forgetting. Task complexity (Nguyen et al., 2019) is positively cor- related with total error observed, but it is unknown how we should arrange tasks in sequence to reduce forgetting. We conduct an empirical study to un- derstand how factors like network architecture and capacity affect forgetting. In this work, we design experiments to empiri- cally address the following research questions: RQ1: Do some neural architectures forget more than others? …

Content truncated to 500 words