Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2542 - 2550 July 10-15, 2022 ©2022 Association for Computational Linguistics Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning Vishakh Padmakumar 1∗ Leonard Lausen2 Miguel Ballesteros3 Sheng Zha2 He He12 George Karypis2 1New York University, 2AWS AI, 3AWS AI Labs vishakh@nyu.edu {lausen, ballemig, zhasheng, hehea, gkarypis}@amazon.com Abstract Recent work has found that multi-task train- ing with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost. 1 Introduction Following the wide success of unsupervised lan- guage model pre-training (Devlin et al., 2019; Liu et al., 2019b; Lewis et al., 2019), recent work on transfer learning has shown that additional super- vised multi-task training further improves perfor- mance on various downstream NLP tasks (Raffel et al., 2019; Khashabi et al., 2020; Aghajanyan et al., 2021). There are two distinct ways in which the supervised data has been used: increasing the scale of the multi-task step to incorporate more tasks (Khashabi et al., 2020; Aghajanyan et al., 2021) and developing task similarity metrics to incorporate tasks related to the target task (Pruk- sachatkun et al., 2020; Vu et al., 2020). Aghajanyan et al. (2021) show that a multi-task training step, or pre-finetuning step, with a suffi- ciently large, diverse set of tasks is an effective task- agnostic second stage of model pre-training before finetuning on target tasks. In particular, they find that using a large number of tasks (e.g., roughly 15 ∗ Work done during summer internship at AWS AI. tasks) is crucial in achieving good downstream per- formance, while pre-finetuning with fewer tasks causes a small performance drop. Meanwhile, work on task transferability (Vu et al., 2020; Pruk- sachatkun et al., 2020) has shown that the choice of individual intermediate tasks significantly affects downstream fine-tuning performance—predicting the transfer is challenging and there is high vari- ance depending on the choice of intermediate task. This motivates us to ask the question if pre- finetuning on a small group of tasks related to the target tasks can obtain comparable performance to large-scale multi-task training. In this work, we present an empirical study to answer this question by extending the task transferability experiment to groups of tasks. We follow the two-step experimental pipeline from Aghajanyan et al. (2021), where a pre-trained model is first pre-finetuned on a set of …

Content truncated to 500 words