Prototypical Verbalizer for Prompt-based Few-shot Tuning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7014 - 7024 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Prototypical Verbalizer for Prompt-based Few-shot Tuning Ganqu Cui1,2, Shengding Hu1,2, Ning Ding1,2, Longtao Huang6, Zhiyuan Liu1,2,3,4,5∗ 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Beijing National Research Center for Information Science and Technology 3Institute Guo Qiang, Tsinghua University, Beijing, China 4International Innovation Center of Tsinghua University, Shanghai, China 5Beijing Academy of Artificial Intelligence 6Alibaba Group cgq19@mails.tsinghua.edu.cn Abstract Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tun- ing wraps the input text into a cloze question. To make predictions, the model maps the out- put words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains chal- lenging. In this work, we propose the proto- typical verbalizer (ProtoVerb) which is built directly from training data. Specifically, Pro- toVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the proto- types summarize training instances and are able to enclose rich class-level semantics. We con- duct experiments on both topic classification and entity typing tasks, and the results demon- strate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surpris- ingly, ProtoVerb consistently boosts prompt- based tuning even on untuned PLMs, indi- cating an elegant non-tuning way to utilize PLMs. Our codes are avaliable at https: //github.com/thunlp/OpenPrompt. 1 Introduction The massive-scale pre-trained language models (PLMs) (Han et al., 2021a) have been proven to be backbones for solving a variety of NLP tasks (Kowsari et al., 2019; Rajpurkar et al., 2016). To further adapt these PLMs to downstream tasks such as classification, traditional approaches fine- tune the language models through an extra classi- fier (Howard and Ruder, 2018). However, when task-specific data is limited (Bragg et al., 2021), training the extra classifier effectively is challeng- ing due to the gap between pre-training tasks (e.g., ∗ Corresponding author: Z.Liu (liuzy@tsinghua.edu.cn) A [MASK] news: Tokyo Olympic Daily Preview, July 26th. sports technology Manual Verbalizer Labels Template Input Tech Sports Soft Verbalizer Search-based Verbalizer key … with gym software Verbalizer Figure 1: Illustration of three verbalizer construction methods. masked language modeling) and fine-tuning tasks (e.g., classification and regression). This gap im- pedes the fast adaptation of PLMs to downstream tasks. Recently, prompt-based tuning (Schick and Schütze, 2021; Liu et al., 2021) has risen to be a powerful way for few-shot learning by bridging the gap between the pre-training stage and down- stream task stage. In prompt-based tuning, the in- put texts are wrapped with task-specific templates to re-formalize the original task as a cloze-style task. For example, in topic classification task, we can use template “<text> This topic is about [MASK]”, where <text> is the placeholder for input sentences. The PLMs are asked to infer …

Content truncated to 500 words