Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3631 - 3643 July 10-15, 2022 ©2022 Association for Computational Linguistics PROMPT WAYWARDNESS: The Curious Case of Discretized Interpretation of Continuous Prompts Daniel Khashabi‡ Xinxi Lyu† Sewon Min† Lianhui Qin† Kyle Richardson‡ Sean Welleck†‡ Hannaneh Hajishirzi†‡ Tushar Khot‡ Ashish Sabharwal‡ Sameer Singh‡♥ Yejin Choi†‡ †University of Washington ‡Allen Institute for AI ♥University of California-Irvine Abstract Fine-tuning continuous prompts for target tasks has recently emerged as a compact alter- native to full model fine-tuning. Motivated by these promising results, we investigate the fea- sibility of extracting a discrete (textual) inter- pretation of continuous prompts that is faithful to the problem they solve. In practice, we ob- serve a “wayward” behavior between the task solved by continuous prompts and the nearest neighbor discrete projections of these prompts: One can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contra- dictory task) and simultaneously being within a very small (2%) margin of the best contin- uous prompt of the same size for the task. We provide intuitions behind this odd and sur- prising behavior, as well as extensive empir- ical analyses quantifying the effect of design choices. For instance, larger models exhibit higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop of accuracy. These find- ings have important implications relating to the difficulty of faithfully interpreting contin- uous prompts and their generalization across models and tasks, providing guidance for fu- ture progress in prompting language models. 1 Introduction Recent work has shown the surprising power of continuous prompts to language models (LMs) for controlled generation and for solving a wide range of tasks (Li and Liang, 2021; Lester et al., 2021; Min et al., 2022). Despite these successes, the resulting continuous prompts are not easy to inter- pret (Shin et al., 2020). Is it possible to come up with meaningful discrete (textual) interpretations of continuous prompts, especially ones that provide a faithful explanation of the prompt’s behavior? Towards addressing this question, we propose and investigate the Prompt Waywardness hypoth- esis (§3.2), a surprising disconnect between the LM task prompts for sentiment classification Input Output " Fl i p t he sent i ment . " " Gener at e a quest i on. " " pr i nt ( " Hel l o Wor l d! " ) discrete projection " A gr eat f ant asy movi e" . . . Figure 1: We show that one can find accurate contin- uous prompts (that do well on a given task, e.g., sen- timent classification here) such that they can be pro- jected to any arbitrary text, such as the definition of a different task (e.g., generating a question) or even an irrelevant statement (e.g., a piece of code) — suggest- ing a disconnect between the outcome of continuous prompts …

Content truncated to 500 words