Adaptable Adapters

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3742 - 3753 July 10-15, 2022 ©2022 Association for Computational Linguistics Adaptable Adapters Nafise Sadat Moosavi1∗, Quentin Delfosse3, Kristian Kersting2,3, Iryna Gurevych2,4 1 Department of Computer Science, The University of Sheffield 2 Hessian Center for AI (hessian.AI) 3 AI & Machine Learning Lab, 4 Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt https://www.ukp.tu-darmstadt.de Abstract State-of-the-art pretrained NLP models con- tain a hundred million to trillion parameters. Adapters provide a parameter-efficient alterna- tive for the full finetuning in which we can only finetune lightweight neural network layers on top of pretrained weights. Adapter layers are initialized randomly. However, existing work uses the same adapter architecture—i.e., the same adapter layer on top of each layer of the pretrained model—for every dataset, regardless of the properties of the dataset or the amount of available training data. In this work, we introduce adaptable adapters that contain (1) learning different activation functions for dif- ferent layers and different input data, and (2) a learnable switch to select and only use the ben- eficial adapter layers. We show that adaptable adapters achieve on-par performances with the standard adapter architecture while using a con- siderably smaller number of adapter layers. In addition, we show that the selected adapter ar- chitecture by adaptable adapters transfers well across different data settings and similar tasks. We propose to use adaptable adapters for de- signing efficient and effective adapter architec- tures. The resulting adapters (a) contain about 50% of the learning parameters of the stan- dard adapter and are therefore more efficient at training and inference, and require less stor- age space, and (b) achieve considerably higher performances in low-data settings.1 1 Introduction Recent improvements in NLP are heavily skewed towards using larger pretrained models (Roberts et al., 2020) and given their considerably better performances, using them is becoming unavoid- able (Kaplan et al., 2020). Their improvements, however, come at the cost of significant computa- tional resources at training and inference times. For ∗The work has been mostly carried out during the employ- ment at the UKP Lab, TU Darmstadt. 1The code is available at https://github.com/ UKPLab/adaptable-adapters. instance, the number of parameters in recent pre- trained models can vary from 110M in BERT-base (Devlin et al., 2019) to 11 billion in T0 (Sanh et al., 2022) to trillion parameters in Switch Transformers (Fedus et al., 2021). Using such models for each downstream application requires a vast amount of storage, training, and inference computation budget that is not accessible to every user. Instead of fine-tuning these massive numbers of parameters for each downstream task, we can use adapter architectures (Houlsby et al., 2019; Pfeiffer et al., 2020). Adapters are lightweight neural net- work layers that are added on top of each layer of the pretrained model. As opposed to the standard model fine-tuning, in which all layers are fine-tuned for the target task, adapter-based tuning freezes …

Content truncated to 500 words