AdapterFusion: Non-Destructive Task Composition for Transfer Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 487–503 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 487 AdapterFusion: Non-Destructive Task Composition for Transfer Learning Jonas Pfeiffer1, Aishwarya Kamath2, Andreas R¨uckl´e1, Kyunghyun Cho2,3, Iryna Gurevych1 1Ubiquitous Knowledge Processing Lab (UKP Lab), Technical University of Darmstadt 2New York University 3CIFAR Associate Fellow pfeiffer@ukp.tu-darmstadt.de Abstract Sequential fine-tuning and multi-task learn- ing are methods aiming to incorporate knowl- edge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcom- ings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowl- edge from multiple tasks. First, in the knowl- edge extraction stage we learn task specific pa- rameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composi- tion step. We show that by separating the two stages, i.e., knowledge extraction and knowl- edge composition, the classifier can effectively exploit the representations learned from mul- tiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 di- verse NLU tasks, and find that it effectively combines various types of knowledge at differ- ent layers of the model. We show that our ap- proach outperforms traditional strategies such as full fine-tuning as well as multi-task learn- ing. Our code and adapters are available at AdapterHub.ml. 1 Introduction The most commonly used method for solving NLU tasks is to leverage pretrained models, with the dominant architecture being a transformer (Vaswani et al., 2017), typically trained with a language modelling objective (Devlin et al., 2019; Radford et al., 2018; Liu et al., 2019b). Transfer to a task of interest is achieved by fine-tuning all the weights of the pretrained model on that single task, often yielding state-of-the-art results (Zhang and Yang, 2017; Ruder, 2017; Howard and Ruder, 2018; Peters et al., 2019). However, each task of in- terest requires all the parameters of the network to be fine-tuned, which results in a specialized model for each task. Feed Forward Multi-Head Attention Add & Norm Add & Norm Add & Norm AdapterFusion Adapter Figure 1: AdapterFusion architecture inside a trans- former (Vaswani et al., 2017). The AdapterFusion com- ponent takes as input the representations of multiple adapters trained on different tasks and learns a parame- terized mixer of the encoded information. There are two approaches for sharing informa- tion across multiple tasks. The first consists of starting from the pretrained language model and sequentially fine-tuning on each of the tasks one by one (Phang et al., 2018). However, as we subse- quently fine-tune the model weights on new tasks, the problem of catastrophic forgetting (McCloskey and Cohen, 1989; French, 1999) can arise, which results in loss of knowledge already learned from all previous tasks. This, together with the non- trivial decision of the order of tasks in which to fine-tune the model, hinders the effective transfer of knowledge. Multi-task learning (Caruana, 1997; Zhang and Yang, 2017; Liu et al., 2019a) …

Content truncated to 500 words