AdapterDrop: On the Efficiency of Adapters in Transformers

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930–7946 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7930 AdapterDrop: On the Efficiency of Adapters in Transformers Andreas Rücklé∗and Gregor Geigle and Max Glockner, Tilman Beck and Jonas Pfeiffer and Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP) Department of Computer Science, Technische Universität Darmstadt www.ukp.tu-darmstadt.de Abstract Transformer models are expensive to fine-tune, slow for inference, and have large storage re- quirements. Recent approaches tackle these shortcomings by training smaller models, dy- namically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that Adap- terDrop can dynamically reduce the compu- tational overhead when performing inference over multiple tasks simultaneously, with mini- mal decrease in task performances. We further prune adapters from AdapterFusion, which im- proves the inference efficiency while maintain- ing the task performances entirely. 1 Introduction While transfer learning has become the go-to method for solving NLP tasks (Pan and Yang, 2010; Torrey and Shavlik, 2010; Ruder, 2019; Howard and Ruder, 2018; Peters et al., 2018), transformer- based models are notoriously deep requiring mil- lions or even billions of parameters (Radford et al., 2018; Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Brown et al., 2020). This results in slow inference and large storage requirements. At least three independent lines of research have recently evolved to tackle these shortcom- ings. (1) Smaller and faster models that are ei- ther distilled or trained from scratch (Sanh et al., 2019; Sun et al., 2020; Bai et al., 2021; Wang et al., 2020). (2) Robustly trained transformers in which the model depth can be reduced at run-time, thereby decreasing inference time dynamically (Fan et al., 2020; Elbayad et al., 2020; Xin et al., 2020; Hou et al., 2020). (3) Adapters, which, instead of fully fine-tuning the model, only train a newly intro- duced set of weights at every layer, thereby sharing ∗Work done prior to joining Amazon. the majority of parameters between tasks (Houlsby et al., 2019; Bapna and Firat, 2019; Pfeiffer et al., 2020a). Adapters have been shown to work well for machine translation (Bapna and Firat, 2019), cross-lingual transfer (Pfeiffer et al., 2020b, 2021b; Üstün et al., 2020; Vidoni et al., 2020; Ansell et al., 2021), community QA (Rücklé et al., 2020), and task composition for transfer learning (Stickland and Murray, 2019; Pfeiffer et al., 2021a; Lauscher et al., 2020; Wang et al., 2021; Poth et al., 2021). Despite their recent popularity, the computational efficiency of adapters has not been explored beyond parameter efficiency. We close this gap and establish the computa- tional efficiency of two adapter architectures at training and inference time. We investigate differ- ent strategies to further improve the efficiency of adapter-based models by incorporating ideas from all three directions mentioned above. Our strategies rely on dropping out adapters from transformers, at …

Content truncated to 500 words