Exploring and Predicting Transferability across NLP Tasks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7882–7926, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7882 Exploring and Predicting Transferability across NLP Tasks Tu Vu1⋆ Tong Wang2 Tsendsuren Munkhdalai2 Alessandro Sordoni2 Adam Trischler2 Andrew Mattarella-Micke3 Subhransu Maji1 Mohit Iyyer1 University of Massachusetts Amherst1 Microsoft Research Montreal2 Intuit AI3 {tuvu,smaji,miyyer}@cs.umass.edu {tong.wang,tsendsuren.munkhdalai}@microsoft.com {alsordo,adam.trischler}@microsoft.com andrew mattarella-micke@intuit.com Abstract Recent advances in NLP demonstrate the effec- tiveness of training large-scale language mod- els and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve per- formance? In this paper, we conduct an ex- tensive study of the transferability between 33 NLP tasks across three broad classes of prob- lems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than pre- viously thought, especially when target task data is scarce, and can improve performance even with low-data source tasks that differ sub- stantially from the target task (e.g., part-of- speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most trans- ferable source tasks for a given target task, and we validate their effectiveness in exper- iments controlled for source and target data size. Overall, our experiments reveal that fac- tors such as data size, task and domain simi- larity, and task complexity all play a role in determining transferability. 1 Introduction With the advent of methods such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019), the dominant paradigm for developing NLP models has shifted to transfer learning: first, pretrain a large language model, and then fine-tune it on the target dataset. Prior work has explored whether fine-tuning on intermediate source tasks before the target task can further improve this pipeline (Phang et al., 2018), but the conditions for successful trans- fer remain opaque, and choosing arbitrary source tasks can even adversely impact downstream per- formance (Wang et al., 2019b). Our work has two ⋆ Part of this work was done during an internship at Microsoft Research. 3. fine-tune BERT on selected source task 4. fine-tune the resulting model on target task 1. given a target task of interest, compute a task embedding from BERT’s layer-wise gradients 2. identify the most similar source task embedding from a precomputed library SQuAD SST2 DROP MNLI QNLI POS-PTB CCG WikiHop WikiHop Target task Figure 1: A demonstration of our task embedding pipeline. Given a target task, we first compute its task embedding and then identify the most similar source task embedding (in this example, WikiHop) from a pre- computed library via cosine similarity. Finally, we per- form intermediate fine-tuning of BERT on the selected source task before fine-tuning on the target task.1 main contributions: (1) we perform a large-scale empirical study across 33 different datasets to shed light on the transferability between NLP tasks, and (2) we develop task embeddings to predict which source tasks to use for a given target task. Our …

Content truncated to 500 words