Do Prompt-Based Models Really Understand the Meaning of Their Prompts?

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300 - 2344 July 10-15, 2022 ©2022 Association for Computational Linguistics Do Prompt-Based Models Really Understand the Meaning of Their Prompts? Albert Webson1,2 and Ellie Pavlick1 {albert_webson, ellie_pavlick}@brown.edu 1Department of Computer Science, Brown University 2Department of Philosophy, Brown University Abstract Recently, a boom of papers has shown ex- traordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help mod- els to learn faster in the same way that hu- mans learn faster when provided with task in- structions expressed in natural language. In this study, we experiment with over 30 prompt templates manually written for natural lan- guage inference (NLI). We find that models can learn just as fast with many prompts that are intentionally irrelevant or even pathologi- cally misleading as they do with instructively “good” prompts. Further, such patterns hold even for models as large as 175 billion parame- ters (Brown et al., 2020) as well as the recently proposed instruction-tuned models which are trained on hundreds of prompts (Sanh et al., 2021). That is, instruction-tuned models of- ten produce good predictions with irrelevant and misleading prompts even at zero shots. In sum, notwithstanding prompt-based models’ impressive improvement, we find evidence of serious limitations that question the degree to which such improvement is derived from mod- els understanding task instructions in ways analogous to humans’ use of task instructions. 1 Introduction Suppose a human is given two sentences: “No weapons of mass destruction found in Iraq yet.” and “Weapons of mass destruction found in Iraq.” They are then asked to respond 0 or 1 and receive a reward if they are correct. In this setup, they would likely need a large number of trials and errors be- fore figuring out what they are really being re- warded to do. This setup is akin to the pretrain-and- fine-tune setup which has dominated NLP in recent years, in which models are asked to classify a sen- tence representation (e.g., a CLS token) into some ∗Unabridged version available on arXiv. Code, interactive figures, and statistical test results available at https://github. com/awebson/prompt_semantics arbitrary dimensions of a one-hot vector. In con- trast, suppose a human is given a prompt such as: Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “ Given that “no weapons of mass destruction found in Iraq yet.”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ ”, is it definitely correct that “ …

Content truncated to 500 words