RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369 - 3391 December 7-11, 2022 ©2022 Association for Computational Linguistics RLPROMPT: Optimizing Discrete Text Prompts with Reinforcement Learning Mingkai Deng1∗, Jianyu Wang2∗, Cheng-Ping Hsieh2∗, Yihan Wang2, Han Guo1, Tianmin Shu3, Meng Song2, Eric P. Xing1,4,5, Zhiting Hu2 1Carnegie Mellon University, 2UC San Diego, 3MIT, 4Mohamed bin Zayed University of Artificial Intelligence, 5Petuum Inc. {mingkaid,hanguo}@cs.cmu.edu, {jiw102,c2hsieh,yiw007,zhh019}@ucsd.edu Abstract Prompting has shown impressive success in en- abling large pre-trained language models (LMs) to perform diverse NLP tasks, especially with only few downstream data. Automatically find- ing the optimal prompt for each task, how- ever, is challenging. Most existing work re- sorts to tuning soft prompts (e.g., embeddings) which fall short of interpretability, reusabil- ity across LMs, and applicability when gra- dients are not accessible. Discrete prompts, on the other hand, are difficult to optimize, and are often created by “enumeration (e.g., paraphrasing)-then-selection” heuristics that do not explore the prompt space systematically. This paper proposes RLPROMPT, an efficient discrete prompt optimization approach with re- inforcement learning (RL). RLPROMPT formu- lates a parameter-efficient policy network that generates the optimized discrete prompt after training with reward. To harness the complex and stochastic reward signals from the large LM environment, we incorporate effective re- ward stabilization that substantially enhances training efficiency. RLPROMPT is flexibly ap- plicable to different types of LMs, such as masked (e.g., BERT) and left-to-right models (e.g., GPTs), for both classification and genera- tion tasks. Experiments on few-shot classifica- tion and unsupervised text style transfer show superior performance over a wide range of ex- isting fine-tuning or prompting methods. Inter- estingly, the resulting optimized prompts are often ungrammatical gibberish text; and surpris- ingly, those gibberish prompts are transferrable between different LMs to retain significant per- formance, indicating that LM prompting may not follow human language patterns.1 1 Introduction Prompting has emerged as a promising approach to solving a wide range of NLP problems using large ∗Equal contribution 1Code available at https://github.com/mingkaid/ rl-prompt pre-trained language models (LMs), including left- to-right models such as GPTs (Radford et al., 2019; Brown et al., 2020) and masked LMs such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), etc. Compared to conventional fine-tuning that expensively updates the massive LM parameters for each downstream task, prompting concatenates the inputs with an additional piece of text that steers the LM to produce the desired outputs. A key question with prompting is how to find the optimal prompts to improve the LM’s performance on various tasks, often with only a few training examples. One of the most popular scheme is to tune soft prompts (i.e., continuous embedding vectors) as they are amenable to gradient descent (Li and Liang, 2021; Vu et al., 2021; Gu et al., 2021; Liu et al., 2021d; Mokady et al., 2021; Qian et al., 2022; An et al., 2022, etc.). However, the result- ing prompts are, by their nature, hard for humans to understand (Khashabi et al., 2021; Lester et al., …

Content truncated to 500 words