Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5698–5711 July 9-14, 2023 ©2023 Association for Computational Linguistics Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts Mohna Chakraborty∗, Adithya Kulkarni*, and Qi Li Department of Computer Science, Iowa State University {mohnac, aditkulk, qli}@iastate.edu Abstract Recent studies have demonstrated that natural- language prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task. Specifically, these methods utilize few-shot learning settings to fine- tune the sentiment classification model using manual or automatically generated prompts. However, the performance of these methods is sensitive to the perturbations of the utilized prompts. Furthermore, these methods depend on a few labeled instances for automatic prompt generation and prompt ranking. This study aims to find high-quality prompts for the given task in a zero-shot setting. Given a base prompt, our proposed approach automatically generates multiple prompts similar to the base prompt employing positional, reasoning, and paraphrasing techniques and then ranks the prompts using a novel metric. We empirically demonstrate that the top-ranked prompts are high-quality and significantly outperform the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task. 1 Introduction The recent advance of large language models such as ChatGPT (ChatGPT, 2022), GPT-3 (Brown et al., 2020), and T5 (Raffel et al., 2020) has shown an astounding ability to understand natural languages. These pre-trained models can conduct various Natural Language Processing (NLP) tasks under the zero/few-shot settings using natural language instructions (i.e., prompts) when no or a few training samples exist. The prompts play crucial roles in these scenarios. The prompts can be generated manually or automatically (Schick and Schütze, 2021; Gao et al., 2021; Gu et al., 2022; Wang et al., 2022). The manual prompts are handcrafted based on the ∗equal contribution user’s intuition of the task (Schick and Schütze, 2021; Gao et al., 2021). Humans can easily write prompts, but the manual prompts are likely to be suboptimal since the language models may understand the instruction differently from humans. Prior studies have also shown that the performance of the language models is sensitive to the choice of prompts. For example, (Gao et al., 2021; Jiang et al., 2020) have shown that the performance is sensitive to the choice of certain words in the prompts and the position of the prompts. Due to the sensitivity and the potential misunderstanding of the instruction, manual prompts tend to suffer from poor performance under zero-shot settings. The language models tend to understand human intentions better when used with a small amount of training data. Therefore, the model can improve significantly under few-shot settings. To address the problems of manual prompts, some studies (Jiang et al., 2020; Gao et al., 2021) further propose to generate prompts automatically following few-shot settings. These models utilize generative language models, such as the T5 model, to write automatic prompts using small training data from the task. Some studies (Shin et al., …

Content truncated to 500 words