Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5231 Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work? Yada Pruksachatkun1∗ Jason Phang1∗ Haokun Liu1∗ Phu Mon Htut1∗ Xiaoyi Zhang1 Richard Yuanzhe Pang1 Clara Vania1 Katharina Kann2 Samuel R. Bowman1 1New York University 2University of Colorado Boulder {yp913,bowman}@nyu.edu Abstract While pretrained models such as BERT have shown large gains across natural language un- derstanding tasks, their performance can be improved by further training the model on a data-rich intermediate task, before fine-tuning it on a target task. However, it is still poorly understood when and why intermediate-task training is beneficial for a given target task. To investigate this, we perform a large-scale study on the pretrained RoBERTa model with 110 intermediate–target task combinations. We further evaluate all trained models with 25 probing tasks meant to reveal the specific skills that drive transfer. We observe that intermediate tasks requiring high-level infer- ence and reasoning abilities tend to work best. We also observe that target task performance is strongly correlated with higher-level abil- ities such as coreference resolution. How- ever, we fail to observe more granular corre- lations between probing and target task per- formance, highlighting the need for further work on broad-coverage probing benchmarks. We also observe evidence that the forgetting of knowledge learned during pretraining may limit our analysis, highlighting the need for further work on transfer learning methods in these settings. 1 Introduction Unsupervised pretraining—e.g., BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019b)—has recently pushed the state of the art on many nat- ural language understanding tasks. One method of further improving pretrained models that has been shown to be broadly helpful is to first fine- tune a pretrained model on an intermediate task, before fine-tuning again on the target task of inter- est (Phang et al., 2018; Wang et al., 2019a; Clark et al., 2019a; Sap et al., 2019), also referred to as ∗Equal contribution. Figure 1: Our experimental pipeline with intermediate- task transfer learning and subsequent fine-tuning on tar- get and probing tasks. STILTs. However, this approach does not always improve target task performance, and it is unclear under what conditions it does. This paper offers a large-scale empirical study aimed at addressing this open question. We per- form a broad survey of intermediate and target task pairs, following an experimental pipeline similar to Phang et al. (2018) and Wang et al. (2019a). This differs from previous work in that we use a larger and more diverse set of intermediate and target tasks, introduce additional analysis-oriented prob- ing tasks, and use a better-performing base model RoBERTa (Liu et al., 2019b). We aim to answer the following specific questions: • What kind of tasks tend to make good inter- mediate tasks across a wide variety of target tasks? • Which linguistic skills does a model learn from intermediate-task training? • Which skills learned …

Content truncated to 500 words