ASCM: An Answer Space Clustered Prompting Method without Answer Engineering

Findings of the Association for Computational Linguistics: ACL 2022, pages 2455 - 2469 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ASCM: An Answer Space Clustered Prompting Method without Answer Engineering Zhen Wang1,3, Yating Yang1,2,3∗, Xi Zhou1,2,3∗, Bo Ma1,2,3, Lei Wang1,2,3, Rui Dong1,2,3, Azmat Anwar1,3 1Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi, China 2University of Chinese Academy of Sciences, Beijing, China 3Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi, China {wang_zhen,yangyt,zhouxi,mabo,wanglei,dongrui,azmat}@ms.xjb.ac.cn Abstract Prompt-based learning, which exploits knowl- edge from pre-trained language models by pro- viding textual prompts and designing appro- priate answer-category mapping methods, has achieved impressive successes on few-shot text classification and natural language inference (NLI). Because of the diverse linguistic expres- sion, there exist many answer tokens for the same category. However, both manual answer design and automatic answer search constrain answer space and therefore hardly achieve ideal performance. To address this issue, we propose an answer space clustered prompting model (ASCM) together with a synonym initialization method (SI) which automatically categorizes all answer tokens in a semantic-clustered em- bedding space. We also propose a stable semi- supervised method named stair learning (SL) that orderly distills knowledge from better mod- els to weaker models. Extensive experiments demonstrate that our ASCM+SL significantly outperforms existing state-of-the-art techniques in few-shot settings.1 1 Introduction Pre-trained language models (PLMs, Vaswani et al., 2017; Devlin et al., 2019; Qiu et al., 2020; Lewis et al., 2020; Clark et al., 2020) have shown a great impact on natural language processing (NLP) tasks. By adding task-specific head and fine-tuning on labeled corpora, PLMs surpass conventional fully supervised learning paradigm and come into being a “pre-train, fine-tune” paradigm (Sun et al., 2019). However, Radford et al. (2019) demonstrate PLMs can perform downstream tasks without any additional data and modification, which reveals PLMs have the potential for knowledge exploration. ∗ Corresponding author 1Our implementation is publicly available at https:// github.com/miaomiao1215/ASCM. PLM __ news: France won the 2018 World Cup __ news: It was the12th title for the Lakers. __ news: Lin Dan won the championship. semantic cluster module Soccer NBA Sports Finance Business Commerce Soccer NBA Sports Finance Business Commerce Emask class E Sports: 0.8 Sports: 0.8 Sports: 0.9 Business: 0.2 Business: 0.2 Business: 0.1 Sports CE L France won the 2018 World Cup It was the12th title for the Lakers. Lin Dan won the championship. : X ˆ: X fprompt   Figure 1: Illustration of ASCM. Given textual input x, ASCM adds task-specific prompt pattern to X and pre- dicts masked token embedding by PLM encoder. Then semantic cluster module (SCM) transforms token em- bedding to a semantic-clustered embedding space. Fi- nally, similarities between Emask and categorized clus- ter centers decide the category of x. And Petroni et al. (2019) find that BERT contains relational knowledge, factual knowledge and can be applied to QA tasks without fine-tuning. Recently, prompting methods (Liu et al., 2021), which reformulate downstream tasks with task- specific textual prompts, is proved successful in 2455 many …

Content truncated to 500 words