On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2208–2222 August 1–6, 2021. ©2021 Association for Computational Linguistics 2208 On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation Ruidan He∗1, Linlin Liu∗12†, Hai Ye∗3, Qingyu Tan13†, Bosheng Ding12†, Liying Cheng14†, Jia-Wei Low12†, Lidong Bing1, Luo Si1 1DAMO Academy, Alibaba Group 2Nanyang Technological University 3National University of Singapore 4Singapore University of Technology and Design {ruidan.he,linlin.liu,qingyu.tan}@alibaba-inc.com {bosheng.ding,liying.cheng,jiawei.low}@alibaba-inc.com yeh@comp.nus.edu.sg {l.bing,luo.si}@alibaba-inc.com Abstract Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allow- ing a high degree of parameter sharing. Prior studies have shown that adapter-based tun- ing often achieves comparable results to fine- tuning. However, existing work only focuses on the parameter-efficient aspect of adapter- based tuning while lacking further investiga- tion on its effectiveness. In this paper, we study the latter. We first show that adapter- based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically com- pare the two tuning methods on several down- stream NLP tasks and settings. We demon- strate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates. 1 Introduction Large scale pretrained language models (PrLMs) (Devlin et al., 2019; Liu et al., 2019; Conneau et al., 2020a; Brown et al., 2020) have achieved state-of- the-art results on most natural language processing (NLP) tasks, where fine-tuning has become a dom- inant approach to utilize PrLMs. A standard fine- tuning process copies weights from a PrLM and tunes them on a downstream task, which requires a new set of weights for each task. Adapter-based tuning (Houlsby et al., 2019; Bapna and Firat, 2019) has been proposed as a ∗ Equally Contributed † Linlin, Qingyu, Bosheng, Liying, and Jia-wei are under the Joint PhD Program between Alibaba and their correspond- ing universities. more parameter-efficient alternative. For NLP, adapters are usually light-weight modules inserted between transformer layers (Vaswani et al., 2017). During model tuning on a downstream task, only the parameters of adapters are updated while the weights of the original PrLM are frozen. Hence, adapter-based tuning adds only a small amount of parameters for each task, allowing a high de- gree of parameter-sharing. Though using much less trainable parameters, adapter-based tuning has demonstrated comparable performance with full PrLM fine-tuning (Houlsby et al., 2019; Bapna and Firat, 2019; Stickland and Murray, 2019). Existing work mostly focuses on the parameter- efficient aspect of adapters and attempt to derive useful applications from that, which is still the case in most recent works: R¨uckl´e et al. (2020) explore methods to further improve the parame- ter and computation …

Content truncated to 500 words