AdapterShare: Task Correlation Modeling with Adapter Differentiation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10645 - 10651 December 7-11, 2022 ©2022 Association for Computational Linguistics AdapterShare: Task Correlation Modeling with Adapter Differentiation Zhi Chen1∗, Bei Chen2, Lu Chen1, Kai Yu1, Jian-Guang Lou2 1X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 2Microsoft Research Asia {zhenchi713, chenlusz, kai.yu}@sjtu.edu.cn, {beichen, jlou}@microsoft.com Abstract Thanks to the development of pre-trained lan- guage models, multitask learning (MTL) meth- ods have achieved great success in natural lan- guage understanding. However, current MTL methods pay more attention to task selection or model design to fuse as much knowledge as possible, while the intrinsic task correla- tion is often neglected. It is important to learn sharing strategies among multiple tasks rather than sharing everything. In this paper, we propose AdapterShare, an adapter differenti- ation method to explicitly model task correla- tion among multiple tasks. AdapterShare is automatically learned based on the gradients on tiny held-out validation data. Compared to single-task learning and fully shared MTL methods, our proposed method obtains obvi- ous performance improvements. Compared to the existing MTL method AdapterFusion, AdapterShare achieves an absolute average im- provement of 1.90 points on five dialogue un- derstanding tasks and 2.33 points on NLU tasks. Our implementation is available at https:// github.com/microsoft/ContextualSP. 1 Introduction With the development of transformer-based pre- trained language models (PLMs), natural language understanding (NLU) has made great progress as a downstream task. There are two main ways to lever- age PLMs in NLU tasks. One is the fine-tuning method, which updates the pre-trained language model directly on a target task. The other one is adapters (Rebuffi et al., 2017; Houlsby et al., 2019), which introduces a small number of task- specific parameters on a fixed PLM. When training on the target task, only the introduced parameters are updated. Compared to the fine-tuning method, adapter is memory-efficient, since the introduced parameters are much less than those of the PLM. In this paper, we focus on the approach using adapters. ∗Work done during internship at Microsoft Research Asia. Multi-Head Attention Feed Forward Adapter Adapter layer-wise differentiation Adapter-1 1 1 0 0 0 1 1 0 0 0 0 0 1 1 1 0 0 1 1 1 0 0 1 1 1 task correlation map 𝒕𝟏 Mode Add & Norm Add & Norm Task Group Filter 𝑡! 𝑡" 𝑡# 𝑡$𝑡% 𝑡! 𝑡" 𝑡# 𝑡$ 𝑡% Φ! & Figure 1: The architecture of the adapters with task correlation modeling method. To transfer the knowledge of different tasks, Stickland and Murray (2019) proposed a multitask learning (MTL) method to update the weights of a shared adapter using the weighting of the objective functions of all target tasks. The shared adapter captures the common structure underlying all the target tasks. This is a typical multitask learning method based on an implicit assumption that all tasks benefit from each other, where all parameters of the adapter are shared during multitask train- ing. …

Content truncated to 500 words