What to Pre-Train on? Efficient Intermediate Task Selection

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585–10605 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10585 What to Pre-Train on? Efficient Intermediate Task Selection Clifton Poth, Jonas Pfeiffer, Andreas Rücklé∗, and Iryna Gurevych Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt www.ukp.tu-darmstadt.de Abstract Intermediate task fine-tuning has been shown to culminate in large transfer gains across many NLP tasks. With an abundance of can- didate datasets as well as pre-trained language models, it has become infeasible to experiment with all combinations to find the best transfer setting. In this work, we provide a compre- hensive comparison of different methods for efficiently identifying beneficial tasks for inter- mediate transfer learning. We focus on param- eter and computationally efficient adapter set- tings, highlight different data-availability sce- narios, and provide expense estimates for each method. We experiment with a diverse set of 42 intermediate and 11 target English clas- sification, multiple choice, question answer- ing, and sequence tagging tasks. Our results demonstrate that efficient embedding based methods, which rely solely on the respective datasets, outperform computational expensive few-shot fine-tuning approaches. Our best methods achieve an average Regret@3 of 1% across all target tasks, demonstrating that we are able to efficiently identify the best datasets for intermediate training. 1 1 Introduction Large pre-trained language models (LMs) are con- tinuously pushing the state of the art across various NLP tasks. The established procedure performs self-supervised pre-training on a large text corpus and subsequently fine-tunes the model on a spe- cific target task (Devlin et al., 2019; Liu et al., 2019b). The same procedure has also been ap- plied to adapter-based training strategies, which achieve on-par task performance to full model fine-tuning while being considerably more param- eter efficient (Houlsby et al., 2019) and faster to ∗Contributions made prior to joining Amazon. 1Code released at https://github.com/ Adapter-Hub/efficient-task-transfer. train (Rücklé et al., 2021).2 Besides being more ef- ficient, adapters are also highly modular, enabling a wider range of transfer learning techniques (Pfeif- fer et al., 2020b, 2021a,b; Üstün et al., 2020; Vidoni et al., 2020; Rust et al., 2021; Ansell et al., 2021). Extending upon the established two-step learn- ing procedure, incorporating intermediate stages of knowledge transfer can yield further gains for fully fine-tuned models. For instance, Phang et al. (2018) sequentially fine-tune a pre-trained language model on a compatible intermediate task before target task fine-tuning. It has been shown that this is most effective for low-resource target tasks, however, not all task combinations are beneficial and many yield decreased performances (Phang et al., 2018; Wang et al., 2019a; Pruksachatkun et al., 2020). The abundance of diverse labeled datasets as well as the continuous development of new pre-trained LMs calls for methods that efficiently identify in- termediate dataset that benefit the target task. So far, it is unclear how adapter-based ap- proaches behave with intermediate fine-tuning. In the first part of this work, we thus establish that this setup results in similar gains for adapters, as has been shown for …

Content truncated to 500 words