AMAL: Meta Knowledge-Driven Few-Shot Adapter Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10381 - 10389 December 7-11, 2022 ©2022 Association for Computational Linguistics AMAL: Meta Knowledge-Driven Few-Shot Adapter Learning S. K. Hong∗ Samsung SDS s.k.hong@samsung.com Tae Young Jang Samsung SDS tae10.jang@samsung.com Abstract NLP has advanced greatly together with the proliferation of Transformer-based pre-trained language models. To adapt to a downstream task, the pre-trained language models need to be fine-tuned with a sufficient supply of anno- tated examples. In recent years, Adapter-based fine-tuning methods have expanded the appli- cability of pre-trained language models by sub- stantially lowering the required amount of an- notated examples. However, existing Adapter- based methods still fail to yield meaningful results in the few-shot regime where only a few annotated examples are provided. In this study, we present a meta-learning-driven low-rank adapter pooling method, called AMAL, for leveraging pre-trained language models even with just a few data points. We evaluate our method on five text classification benchmark datasets. The results show that AMAL signifi- cantly outperforms previous few-shot learning methods and achieves a new state-of-the-art. 1 Introduction Since Transformer-based (Vaswani et al., 2017) pre-trained language models (PLMs) on massive corpora made a big impact on NLP, fine-tuning PLMs (Devlin et al., 2019; Lan et al., 2019; Liu et al., 2019) has led to large improvements in a va- riety of downstream NLP tasks. Yet, it is still chal- lenging to fine-tune PLMs (Zhang et al., 2020) in the few-shot regime. Recently, Adapters (Houlsby et al., 2019a; Ben Zaken et al., 2022; Fu et al., 2022; Hu et al., 2021) have provided a method of fine-tuning PLMs more efficiently, by tuning some extra weights (the Adapters) while freezing the rest. Nevertheless, existing Adapters still fail to yield significant results in the few-shot regime. Refer to the Appendix table 4 for the performance of the prior Adapters on the few-shot classification problems. ∗corresponding author Since GPT-3 (Brown et al., 2020) was intro- duced, prompt tuning has swept the machine learn- ing community. However, finding proper prompts (Schick and Schütze, 2020) is still a delicate task — requiring labor-intensive manual handcrafting with domain expertise as well as an in-depth understand- ing of the language model’s inner mechanisms. In this paper, we present a cost-effective method for language model fine-tuning that is applicable, without customization, to a variety of language models and Adapter types. We focus on small to mid-sized language models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), ALBERT (Lan et al., 2019), BART (Lewis et al., 2020), or DeBERTa (He et al., 2020), because they are widely deployed in production systems due to their econ- omy and low carbon footprint. In this paper, we propose a meta-knowledge- driven few-shot adapter learning method, called AMAL (Adapter-by-MetA-Learning), based on a novel meta-learning framework, through which meta-level layer-wise adaptation kernels are de- rived in an end-to-end manner. Our design takes inspiration from (Aghajanyan et al., 2020), which proves that the over-parameterized pre-trained lan- guage models actually …

Content truncated to 500 words