Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 793 - 808 December 7-11, 2022 ©2022 Association for Computational Linguistics Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning Yifan Chen1∗ Devamanyu Hazarika2∗ Mahdi Namazifar2 Yang Liu2 Di Jin2† Dilek Hakkani-Tur2 1University of Illinois Urbana-Champaign 2Amazon Alexa AI Abstract Prefix-tuning, or more generally continu- ous prompt tuning, has become an essential paradigm of parameter-efficient transfer learn- ing. Using a large pre-trained language model (PLM), prefix-tuning can obtain strong perfor- mance by training only a small portion of pa- rameters. In this paper, we propose to under- stand and further develop prefix-tuning through the kernel lens. Specifically, we make an anal- ogy between prefixes and inducing variables in kernel methods and hypothesize that prefixes serving as inducing variables would improve their overall mechanism. From the kernel esti- mator perspective, we suggest a new variant of prefix-tuning—inducer-tuning, which shares the exact mechanism as prefix-tuning while leveraging the residual form found in adapter- tuning. This mitigates the initialization issue in prefix-tuning. Through comprehensive em- pirical experiments on natural language under- standing and generation tasks, we demonstrate that inducer-tuning can close the performance gap between prefix-tuning and fine-tuning. 1 Introduction Transfer learning from large pre-trained language models (PLMs) has been the de-facto method to tackle downstream natural language processing (NLP) tasks with proven performance and scalabil- ity (Peters et al., 2018). Among all the adaption techniques, fine-tuning (Howard and Ruder, 2018; Kale and Rastogi, 2020) is predominant for PLMs and maintains the models’ architecture while up- dating all the parameters within. Though power- ful, fine-tuning is considered parameter-inefficient since it results in separate copies of model parame- ters for each task/client after training. With the sizes of PLMs increasing to hundreds of millions (Brown et al., 2020) or even up to tril- ∗Equal contribution. This work was performed while the first author was interning at Amazon Alexa AI. †Correspondence to: Di Jin <djinamzn@amazon.com> lion (Fedus et al., 2021) parameters, the trend mo- tivates a range of parameter-efficient adaptation techniques, including adapter-tuning and prompt- ing, as promising lightweight alternatives to fine- tuning to reduce computational consumption and storage space. Adapter-tuning inserts bottlenecked Multi-layer Perception (MLP) modules between the pre-trained layers of PLMs and tunes only these new parameters for task adaptation (Houlsby et al., 2019; Pfeiffer et al., 2020a). Prompting, instead, aims to adapt the general-purpose PLMs through prompts, whose effectiveness has been shown on a frozen GPT-3 model (Brown et al., 2020). An implicit drawback of the prompt-based adap- tation is the difficulty of searching for the proper prompt. To avoid manually designing the prompts, Shin et al. (2020) propose a search algorithm to find the effective prompt over discrete space of vo- cabularies; prefix-tuning (Li and Liang, 2021) and other concurrent methods (Lester et al., 2021; Liu et al., 2021b,a) further extend the discrete search to continuous prompts, attaining performance close to fine-tuning in some tasks. Despite the effort, there is still a performance gap between “prefix- tuning” and “fine-tuning” in many tasks, …

Content truncated to 500 words