Few-shot Query-Focused Summarization with Prefix-Merging

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3704 - 3714 December 7-11, 2022 ©2022 Association for Computational Linguistics Few-shot Query-Focused Summarization with Prefix-Merging Ruifeng Yuan The Hong Kong Polytechnic University csryuan@comp.polyu.edu.hk Zili Wang Xidian University ziliwang.do@gmail.com Ziqiang Cao ∗ Soochow University zqcao@suda.edu.cn Wenjie Li The Hong Kong Polytechnic University cswjli@comp.polyu.edu.hk Abstract Query-focused summarization has been consid- ered as an important extension for text summa- rization. It aims to generate a concise highlight for a given query. Different from text sum- marization, query-focused summarization has long been plagued by the problem of lacking high-quality large-scale datasets. In this pa- per, we investigate the idea that whether we can integrate and transfer the knowledge of text summarization and question answering to as- sist the few-shot learning in query-focused sum- marization. Here, we propose prefix-merging, a prefix-based pretraining strategy for few- shot learning in query-focused summarization. Drawn inspiration from prefix-tuning, we are allowed to integrate the task knowledge from text summarization and question answering into a properly designed prefix and apply the merged prefix to query-focused summarization. With only a small amount of trainable param- eters, prefix-merging outperforms fine-tuning on query-focused summarization. We further discuss the influence of different prefix designs and propose a visualized explanation for how prefix-merging works. 1 Introduction Text summarization aims to compress the source document(s) into a shorter version that contains its important information. As a classic sub-topic for text summarization, query-focused summarization meets that situation that only a specific aspect of information is needed to be summarized. In other words, it aims to generate a summary based on the source content related to a given query. Hence, this task requires not only to locate relevant content in a passage as question answering (QA) but also to summarize and generate a highlight as text sum- marization. Although text summarization has been widely studied in recent years, there are fewer at- tempts on exploring query-focused summarization 1Corresponding author (Deng et al., 2020; Su et al., 2020; Xu and Lap- ata, 2020a; Su et al., 2021) after the age of neural model. One main reason is the lack of general- ized large-scale datasets. Compared with the easily accessible nature reference summaries such as ti- tles or headlines in text summarization, it is hard to collect large-scale data for query-focused sum- marization. Meanwhile, human-written reference summaries have always been costly. The rapidly developed few-shot learning tech- niques provides potential cues to alleviate the prob- lem of lacking large-scale dataset for query-focused summarization, and knowledge transferring is one of them. In fact, when facing unseen tasks, it is natural for human beings to integrate and trans- fer the knowledge of known tasks to relevant new tasks. Inspired by this, we innovatively propose to decouple the query-focused summarization to two basic tasks, i.e. text summarization and question answering, and transfer the knowledge from these two tasks to query-focused summarization. How- ever, in parameter-based knowledge learning, pre- vious work are usually one-to-one (pre-train then fine-tune (Yosinski et al., …

Content truncated to 500 words