Efficiently Tuned Parameters Are Task Embeddings

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5007 - 5014 December 7-11, 2022 ©2022 Association for Computational Linguistics Efficiently Tuned Parameters are Task Embeddings Wangchunshu Zhou1∗, Canwen Xu2∗, Julian McAuley2 1 ETH Zurich 2 University of California, San Diego 1wangchunshu.zhou@inf.ethz.ch, 2{cxu,jmcauley}@ucsd.edu Abstract Intermediate-task transfer can benefit a wide range of NLP tasks with properly selected source datasets. However, it is computation- ally infeasible to experiment with all interme- diate transfer combinations, making choosing a useful source task a challenging problem. In this paper, we anticipate that task-specific pa- rameters updated in parameter-efficient tuning methods are likely to encode task-specific in- formation. Therefore, such parameters can be predictive for inter-task transferability. Thus, we propose to exploit these efficiently tuned parameters as off-the-shelf task embeddings for the efficient selection of source datasets for intermediate-task transfer. We experiment with 11 text classification tasks and 11 question an- swering tasks. Experimental results show that our approach can consistently outperform exist- ing inter-task transferability prediction methods while being conceptually simple and computa- tionally efficient. Our analysis also reveals that the ability of efficiently tuned parameters on transferability prediction is disentangled with their in-task performance. This allows us to use parameters from early checkpoints as task embeddings to further improve efficiency.1 1 Introduction The pretraining then fine-tuning paradigm (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018, 2019; Brown et al., 2020; Lewis et al., 2020; Raffel et al., 2019) has substantially improved the state-of-the-art on a wide range of natural language processing (NLP) tasks. In this paradigm, we first pretrain a large language model on large-scale cor- pora in a general domain, and then fine-tune the pretrained model to be a task-specific model on the target dataset. In addition to directly trans- ferring from a general pretrained language model, ∗Equal contribution. 1Code available at https://github.com/JetRunner/ TuPaTE. Layer n Layer 1 Layer 2 … Extract Average Train Set Task Dataset Parameter-Efficient Tuning Task Embedding Figure 1: The workflow of using efficiently tuned param- eters as task embeddings. The yellow boxes represent tunable parameters in Transformer layers. prior work (Phang et al., 2018) also shows that intermediate-task transfer, i.e., fine-tuning on in- termediate source tasks before the target task, can further improve target task performance. However, the success of intermediate-task transfer heavily relies on the selection of a proper source dataset while an inappropriate source dataset often leads to performance degradation compared to plain fine- tuning. Therefore, some recent works (Vu et al., 2020; Poth et al., 2021) investigate methods to ef- ficiently predict inter-task transferability without actually trying out all intermediate-task combina- tions. The current state of the art (Vu et al., 2020) on predicting inter-task transferability is built on Task2Vec (Achille et al., 2019), which considers the Fisher information matrix of a model fine- tuned on a task as the “task embedding”, and pre- dicts inter-task transferability by computing the cosine similarity between the task embedding of the source and target tasks. Despite empirically per- forming …

Content truncated to 500 words