Learning to Transfer Prompts for Text Generation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3506 - 3518 July 10-15, 2022 ©2022 Association for Computational Linguistics Learning to Transfer Prompts for Text Generation Junyi Li1,3,4, Tianyi Tang1, Jian-Yun Nie3, Ji-Rong Wen1,2,4 and Wayne Xin Zhao1,4∗ 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3DIRO, Université de Montréal 4Beijing Key Laboratory of Big Data Management and Analysis Methods {lijunyi,steven_tang,jrwen}@ruc.edu.cn nie@iro.umontreal.ca batmanfly@gmail.com Abstract Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning. While, it is challenging to fine-tune PLMs in a data-scarce situation. Therefore, it is non-trivial to develop a general and lightweight model that can adapt to various text generation tasks based on PLMs. To fulfill this purpose, the recent prompt-based learning offers a potential solution. In this paper, we improve this technique and propose a novel prompt-based method (PTG) for text generation in a transferable setting. First, PTG learns a set of source prompts for various source generation tasks and then transfers these prompts as target prompts to perform target generation tasks. To consider both task- and instance-level information, we design an adaptive attention mechanism to derive the target prompts. For each data instance, PTG learns a specific target prompt by attending to highly relevant source prompts. In extensive experiments, PTG yields competitive or better results than fine-tuning methods. We release our source prompts as an open resource, where users can add or reuse them to improve new text generation tasks for future research. Code and data can be avail- able at https://github.com/RUCAIBox/ Transfer-Prompts-for-Text-Generation. 1 Introduction In natural language processing (NLP), text genera- tion is an important research topic that aims to au- tomatically produce understandable text in human language from input data (Li et al., 2022). In recent decades, various approaches have been widely ap- plied to a variety of text generation tasks (Li et al., 2019; Gehring et al., 2017; Li et al., 2021a), espe- cially the emergence of pretrained language models (PLMs) (Li et al., 2021c). By involving large-scale parameters pretrained on massive general corpora, ∗Corresponding author PLMs such as GPT-3 (Brown et al., 2020) have achieved substantial progress in text generation. Through the fine-tuning paradigm, PLMs can adapt to various text generation tasks by directly adjust- ing the model parameters with labelled datasets. However, in real-world scenarios, we are in- evitably confronted with tasks having only limited labelled data (e.g., new domains). It is often diffi- cult to fine-tune text generation models in a data- scarce situation (Chen et al., 2020; Li et al., 2021b). Although the input and output formats are differ- ent for various text generation tasks, these tasks essentially adopt similar learning and generation mechanism (e.g., Seq2Seq (Sutskever et al., 2014)). Furthermore, the success of PLMs sheds light on the possibility of developing general or transferable text generation models. For example, Radford et al. (2019) framed generation tasks as language mod- eling by predicting the …

Content truncated to 500 words