Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 4965–4975 October 12–17, 2022. 4965 Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models Zichun Yu1 Tianyu Gao2 Zhengyan Zhang13 Yankai Lin4 Zhiyuan Liu1356† Maosong Sun1356† Jie Zhou7 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University 2Princeton University 3Beijing National Research Center for Information Science and Technology 4Gaoling School of Artificial Intelligence, Renmin University of China 5International Innovation Center of Tsinghua University, Shanghai, China 6Beijing Academy of Artificial Intelligence 7Pattern Recognition Center, WeChat AI, Tencent Inc yuzc19@mails.tsinghua.edu.cn {liuzy,sms}@tsinghua.edu.cn Abstract Prompting, which casts downstream applica- tions as language modeling tasks, has shown to be sample efficient compared to standard fine- tuning with pre-trained models. However, one pitfall of prompting is the need of manually- designed patterns, whose outcome can be unin- tuitive and requires large validation sets to tune. To tackle the challenge, we propose AutoSeq, a fully automatic prompting method: (1) We adopt natural language prompts on sequence- to-sequence models, enabling free-form gen- eration and larger label search space; (2) We propose label sequences – phrases with indef- inite lengths to verbalize the labels – which eliminate the need of manual templates and are more expressive than single label words; (3) We use beam search to automatically gen- erate a large amount of label sequence candi- dates and propose contrastive re-ranking to get the best combinations. AutoSeq significantly outperforms other no-manual-design methods, such as soft prompt tuning, adapter tuning, and automatic search on single label words; the generated label sequences are even better than curated manual ones on a variety of tasks. Our method reveals the potential of sequence- to-sequence models in few-shot learning and sheds light on a path to generic and automatic prompting. The source code of this paper can be obtained from https://github.com/ thunlp/Seq2Seq-Prompt. 1 Introduction Among ways of adapting pre-trained language mod- els (Devlin et al., 2019; Raffel et al., 2020) to Part of the work was done while Yankai Lin was working at Tencent. † Corresponding authors A wonderful movie. It was[MASK]. Highly recommended. A waste of time. great terrible Single label word Label sequence A wonderful movie.[MASK] Sequence-to-sequence model MLM or Sequence-to-sequence model Manually designed template No template Figure 1: Single label words vs label sequences. Label sequences are more expressive and eliminate the need of manually-designed templates. downstream applications, prompting, which uses a natural language prompt to reformulate tasks as cloze questions, has shown to be especially ef- fective (Brown et al., 2020; Schick and Schütze, 2021a,b; Gao et al., 2021). For example, in sen- timent classification, prompting appends a tem- plate “It was [MASK]” to the original input, and defines “great” and “terrible” as the label words, whose probabilities at [MASK] indicate the prob- abilities of the positive and negative sentiment la- bels. Prompting possesses better sample efficiency and performs significantly better than standard fine- tuning in the low resource case. However, the prompting performance is highly sensitive to the prompt choice, whose effectiveness needs abundant validation data to evaluate …

Content truncated to 500 words