An Evaluation of Progressive Neural Networksfor Transfer Learning in Natural Language Processing

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1376–1381 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1376 An Evaluation of Progressive Neural Networks for Transfer Learning in Natural Language Processing Gerhard Hagerer1†, Abdul Moeed1†, Sumit Dugar1†, Sarthak Gupta1†, Mainak Ghosh1† Hannah Danner1 Oliver Mitevski2, Andreas Nawroth2, Georg Groh1 1 Technical University of Munich, 2 Munich Re † These authors contributed equally. gerhard.hagerer@tum.de Abstract A major challenge in modern neural networks is the utilization of previous knowledge for new tasks in an effective manner, otherwise known as transfer learning. Fine-tuning, the most widely used method for achieving this, suffers from catastrophic forgetting. The problem is often exacerbated in natural language processing (NLP). In this work, we assess progressive neural networks (PNNs) as an alternative to fine-tuning. The evaluation is based on common NLP tasks such as sequence labeling and text classification. By gauging PNNs across a range of architectures, datasets, and tasks, we observe improvements over the baselines throughout all experiments. Keywords: Document Classification, Text categorisation, Named Entity Recognition, Opinion Mining / Sentiment Analysis, Statistical and Machine Learning Methods, Other (Transfer Learning) 1. Introduction Transfer learning is the ability of a model to generalize over previously unseen domains and/or tasks in a compe- tent manner. The intuition is to re-use previously learned knowledge effectively when learning new tasks. The most common approaches to transfer learning include fine- tuning and multi-task learning (MTL). The former, where the weights of the already pre-trained layers are re-trained for a new task, performs well for similar tasks (Min et al., 2017) but fails to transfer over unrelated tasks (Mou et al., 2016). The latter adds terms to the objective function for each new task (Rei, 2017), which means re-training the whole model from scratch each time a new task is added (Chen et al., 2017). A major problem faced by traditional transfer learning ap- proaches is catastrophic forgetting (French, 1999) – a phe- nomenon where the model loses performance on previously learned tasks when trained on a new task. Catastrophic for- getting is thoroughly documented in artificial neural net- work literature and a few solutions have been proposed (Kirkpatrick et al., 2016; Awasthi and Sarawagi, 2019). The problem is more prevalent in NLP compared to com- puter vision; the shallow nature of networks used for NLP has been cited as a possible explanation for this discrepancy (Howard and Ruder, 2018). Transfer learning approaches other than fine-tuning and MTL have also been explored for neural architectures (Ho- das et al., 2017; Riemer et al., 2017). One such example - progressive neural networks (PNNs) (Rusu et al., 2016) - offers a novel solution to catastrophic forgetting. The idea is to train multiple networks - one for each new domain/task - that share information learned from previous tasks with each other through lateral connections. PNNs have gained popularity and have already been used for transfer learn- ing in video summarization (Choi et al., 2018) and emotion recognition …

Content truncated to 500 words