PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization

Proceedings of the 29th International Conference on Computational Linguistics, pages 6355–6368 October 12–17, 2022. 6355 PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization Xiaochen Liu†, Yang Gao†∗, Yu Bai†, Jiawei Li†, Yinan Hu†, Heyan Huang† and Boxing Chen †School of Computer Science and Technology, Beijing Institute of Technology {xcliu,gyang,yubai,jwli,ynhu,hhy63}@bit.edu.cn chenboxing@gmail.com Abstract Few-shot abstractive summarization has be- come a challenging task in natural language generation. To support it, we developed a novel soft prompts architecture coupled with a prompt pre-training plus prompt fine-tuning paradigm, which is effective and tunes only extremely light parameters. To meet the struc- ture of the generation models, the soft prompts comprise continuous input embeddings across an encoder and a decoder. Importantly, a new inner-prompt placed in the text is introduced to capture document-level information. The aim is to devote attention to understanding the document that better prompts the model to generate document-related content. In the training process, the prompt pre-training with self-supervised pseudo-data firstly teaches the model basic summarizing capability. Then, with few-shot examples, only the designed lightweight soft prompts are fine-tuned. Ex- perimental results on the CNN/DailyMail and XSum datasets show that our method, with only 0.1% of the parameters, outperforms full-model tuning where all model parameters are tuned. It also surpasses Prompt Tuning by a large margin and delivers competitive results against Prefix- Tuning with 3% of the parameters. 1 Introduction Given the high labor-costs of obtaining quality ab- stractive summaries, few-shot abstractive summa- rization is very demanding and highly challenging. A widely accepted paradigm for almost all NLP tasks is to fine-tune the entire set of parameters for a large pre-trained language model to suit the target task (Liu and Lapata, 2019; Liu et al., 2020). However, the fine-tuning with few-shot exam- ples usually leads to disappointing results, espe- cially with generation tasks like abstractive sum- marization (Fabbri et al., 2020; Yu et al., 2021). The likely outcome is an overfit model. Further, for ∗Corresponding author. every specific task, a large number of pre-trained parameters need to be updated and stored, which is not efficient to use. Pre-trained language models are few-shot learn- ers, i.e., GPT-3 (Brown et al., 2020) that sur- prisingly perform generation tasks from a few examples without any further gradient updates. Although it lacks a rigorously theoretical proof, prompt learning inherits the few-shot property (Li and Liang, 2021; Schick and Schütze, 2020; Jin et al., 2021; Liu et al., 2021). Commonly, this type of learning is considered to retrieve relevant knowl- edge from frozen language models, only tuning continuous prompts to quickly adapt to new tasks with very few examples. More recently, Prompt Tuning (Lester et al., 2021) has received much attention. With large frozen language models (say, >10 billion param- eters), Prompt Tuning simply adds a tunable soft prompt to the input of the encoder, achieving re- sults that are comparable to full-model tuning. Yet, our empirical results, in Section 2, demonstrate that Prompt Tuning for abstractive summariza- tion yields simply abysmal performance. Prefix- Tuning (Li and Liang, …

Content truncated to 500 words