Context-Tuning: Learning Contextualized Prompts for Natural Language Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 6340–6354 October 12–17, 2022. 6340 Context-Tuning: Learning Contextualized Prompts for Natural Language Generation Tianyi Tang1,4, Junyi Li1,3, Wayne Xin Zhao1,4,5 � and Ji-Rong Wen1,2,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3DIRO, Université de Montréal 4Beijing Key Laboratory of Big Data Management and Analysis Methods 5Beijing Academy of Artificial Intelligence, Beijing, 100084, China steventianyitang@outlook.com lijunyi@ruc.edu.cn batmanfly@gmail.com Abstract Recently, pretrained language models (PLMs) have had exceptional success in language gen- eration. To leverage the rich knowledge encoded by PLMs, a simple yet powerful paradigm is to use prompts in the form of either discrete tokens or continuous embed- dings. In existing studies, these prompting methods are typically independent of the in- puts, lacking sufficient consideration of input semantics. To address this issue, we propose a novel continuous prompting approach, called context-tuning, to fine-tuning PLMs for natu- ral language generation. Firstly, the prompts are derived based on the input text to elicit useful knowledge from PLMs for generation. We refer to such prompts as contextualized prompts. Secondly, we use continuous in- verse prompting to improve the process of nat- ural language generation by modeling an in- verse generation process from output to in- put, making the generated text more relevant to the inputs. Furthermore, we utilize a lightweight context-tuning method that fine- tunes only 0.12% of the parameters while maintaining good performance. Our code is publicly available at https://github. com/RUCAIBox/Context-Tuning. 1 Introduction Natural language generation (a.k.a. text genera- tion) aims to produce plausible and readable text in human language from input data (Li et al., 2022). Recently, large-scale pretrained language models (PLMs) such as BART (Lewis et al., 2020) have had exceptional success in language generation. To leverage the encoded knowledge from PLMs, prompting methods have been proposed (Liu et al., 2021), where the original input to PLMs has been extended by prepending discrete tokens or continu- ous embeddings (called prompts). Following this paradigm, this work aims to study how to develop � Corresponding author Title :::::::: Live-action::::::::: medium is inferior to animation medium Static Prompts: Write a story about: Title Contextualized Prompts: p1···k Title pk+1···2k Story I think that live-action works can’t be considered art. They feel more like::::::::::: documentaries or ::::: theater :::: pieces with ::: CGI combined. The superiority of animated works is that they are more abstract and imaginative and characters show more emotion and variety of designs. Table 1: Example inputs (titles) and outputs (stories) of generation dataset CMV. Static prompts are human- written instructions, which are independent of input ti- tles. p1···k and pk+1···2k are contextualized prompts, which are derived conditioned on the input title. The wavy line and underline denote the corresponding in- formation between input and output. more effective prompting methods for text genera- tion based on PLMs. Early methods focused on human-written (dis- crete) prompts by manually constructing task- specific prompt templates (Raffel et al., 2020; Rad- ford et al., 2019), such as “TL;DR:” …

Content truncated to 500 words