Exploring Lottery Prompts for Pre-trained Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15428–15444 July 9-14, 2023 ©2023 Association for Computational Linguistics Exploring Lottery Prompts for Pre-trained Language Models Yulin Chen1∗, Ning Ding1,2∗, Xiaobin Wang3, Shengding Hu2, Hai-Tao Zheng1,4†, Zhiyuan Liu2,5,6† , Pengjun Xie3 1Shenzhen International Graduate School, Tsinghua University 2DCST, Tsinghua University 3Alibaba Group, 4Pengcheng Laboratory, Shenzhen, 5BNRIST, IAI, Tsinghua University 6Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University, Xuzhou {yl-chen21, dingn18}@mails.tsinghua.edu.cn {zheng.haitao}@sz.tsinghua.edu.cn, {liuzy}@tsinghua.edu.cn Abstract Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning. Given the advantage of prompting in the zero-shot setting and the observed performance fluctuation among different prompts, we explore the instance-level prompt and their generalizability. By searching through the prompt space, we first validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM, and such prompt can be obtained at a low cost thanks to the inherent ability of PLMs. Meanwhile, we find that some strong lottery prompts have high performance over the whole training set, and they are equipped with distinguishable linguistic features. Lastly, we attempt to generalize the searched strong lottery prompts to unseen data with prompt ensembling method without any parameter tuning. Experiments are conducted on various types of NLP classification tasks and demonstrate that the proposed method can achieve comparable results with other gradient-free and optimization-free baselines. 1 Introduction Since pre-trained language models (PLMs) became the de-facto standard in modern NLP re- searches (Devlin et al., 2019; Liu et al., 2019; Han et al., 2021a), the pretraining-finetuning paradigm has been prevailing until recent years when models keep scaling (Radford et al., 2019; Brown et al., 2020; Rae et al., 2021) and become too expensive to be optimized. To this end, researchers are ac- tively seeking more effective strategies that require little or even no optimization to harness PLMs. Among these exploratory studies of advanced model adaptation, prompting (Brown et al., ∗ equal contributions † corresponding authors 2020; Schick et al., 2020; Schick and Schütze, 2021a; Gao et al., 2021) is gaining popularity in the community, which uses additional context (prompts) to wrap input instances and trigger desired output tokens. Note that in this paper, the term “prompt” technically refers to the template that wraps the original input. In classification tasks, these tokens are further mapped to particular labels by a verbalizer. Such a paradigm is verified to be effective in a variety of downstream tasks, even when annotations are insufficient. Particularly, empirical evidence shows that coincidental prompts could achieve extraordinary performance in the zero-shot setting, i.e., no training examples are presented. For example, simple manual prompt can achieve an F1 score of over 60% on 46-class entity typing dataset (Ding et al., 2021a) and reaches 73% accuracy on DBpedia with 14 classes (Hu et al., 2021) in the zero-shot setting. Despite the promising performance of prompt- ing, it is often accompanied by drastic …

Content truncated to 500 words