An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Proceedings of NAACL-HLT 2019, pages 2089–2095 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2089 An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models Alexandra Chronopoulou1, Christos Baziotis1, Alexandros Potamianos1,2,3 1School of ECE, National Technical University of Athens, Athens, Greece 2 Signal Analysis and Interpretation Laboratory (SAIL), USC, Los Angeles, USA 3 Behavioral Signal Technologies, Los Angeles, USA el12068@central.ntua.gr, cbaziotis@mail.ntua.gr potam@central.ntua.gr Abstract A growing number of state-of-the-art trans- fer learning methods employ language mod- els pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that ad- dresses the problem of catastrophic forgetting. Specifically, we combine the task-specific op- timization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language reg- ularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pre- training or finetuning separate components of the network and we train our models end-to- end in a single step. We present results on a va- riety of challenging affective and text classifi- cation tasks, surpassing well established trans- fer learning methods with greater level of com- plexity. 1 Introduction Pretrained word representations captured by Lan- guage Models (LMs) have recently become pop- ular in Natural Language Processing (NLP). Pre- trained LMs encode contextual information and high-level features of language, modeling syntax and semantics, producing state-of-the-art results across a wide range of tasks, such as named entity recognition (Peters et al., 2017), machine transla- tion (Ramachandran et al., 2017) and text classifi- cation (Howard and Ruder, 2018). However, in cases where contextual embed- dings from language models are used as additional features (e.g. ELMo (Peters et al., 2018)), results come at a high computational cost and require task-specific architectures. At the same time, ap- proaches that rely on fine-tuning a LM to the task at hand (e.g. ULMFiT (Howard and Ruder, 2018)) depend on pretraining the model on an exten- sive vocabulary and on employing a sophisticated slanted triangular learning rate scheme to adapt the parameters of the LM to the target dataset. We propose a simple and effective transfer learning approach, that leverages LM contextual representations and does not require any elaborate scheduling schemes during training. We initially train a LM on a Twitter corpus and then transfer its weights. We add a task-specific recurrent layer and a classification layer. The transferred model is trained end-to-end using an auxiliary LM loss, which allows us to explicitly control the weighting of the pretrained part of the model and ensure that the distilled knowledge it encodes is preserved. Our contributions are summarized as follows: 1) We show that transfer learning from language models can achieve competitive results, while also being intuitively simple and computationally ef- fective. 2) We address the problem of catastrophic forgetting, by adding an auxiliary LM objective and using an unfreezing method. 3) Our results show that our approach is competitive with more sophisticated transfer learning …

Content truncated to 500 words