PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5245 - 5263 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization Wen Xiao†∗ Iz Beltagy‡ Giuseppe Carenini† Arman Cohan‡§ †University of British Columbia, Vancouver, Canada ‡Allen Institute for AI, Seattle, WA, USA §Paul G. Allen School of Computer Science & Engineering, University of Washington {xiaowen3,carenini}@cs.ubc.ca, {beltagy,armanc}@allenai.org Abstract We introduce PRIMERA, a pre-trained model for multi-document representation with a fo- cus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and ag- gregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outper- forms current state-of-the-art dataset-specific and pre-trained models on most of these set- tings with large margins.1 1 Introduction Multi-Document Summarization is the task of generating a summary from a cluster of re- lated documents. State-of-the-art approaches to multi-document summarization are primarily ei- ther graph-based (Liao et al., 2018; Li et al., 2020; Pasunuru et al., 2021), leveraging graph neural net- works to connect information between the docu- ments, or hierarchical (Liu and Lapata, 2019a; Fabbri et al., 2019; Jin et al., 2020), building inter- mediate representations of individual documents and then aggregating information across. While ef- fective, these models either require domain-specific additional information e.g. Abstract Meaning Representation (Liao et al., 2018), or discourse graphs (Christensen et al., 2013; Li et al., 2020), or use dataset-specific, customized architectures, mak- ing it difficult to leverage pretrained language mod- els. Simultaneously, recent pretrained language models (typically encoder-decoder transformers) ∗Work mainly done during an internship at AI2. 1The code and pre-trained models can be found at https: //github.com/allenai/PRIMER General-Purpose Pre-trained Models T5 BART BigBird LED Task-Specific Pre-trained Models PEGASUS PRIMERA Single-doc Summarization Multi-doc Summarization + Pre-training w/ Entity Pyramid + Model Struct for multi-doc input Figure 1: PRIMERA vs existing pretrained models. have shown the advantages of pretraining and trans- fer learning for generation and summarization (Raf- fel et al., 2020; Lewis et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020). Yet, existing pretrained models either use single-document pretraining ob- jectives or use encoder-only models that do not work for generation tasks like summarization (e.g., CDLM, Caciularu et al., 2021). Therefore, we argue that these pretrained models are not necessarily the best fit for multi-document summarization. Alternatively, we propose a simple pretraining approach for multi-document summa- rization, reducing the need for dataset-specific ar- chitectures and large fine-tuning labeled data (See Figure 1 to compare with other pretrained mod- els). Our method is designed to teach the model to identify and aggregate salient information across a “cluster” of related documents during pretrain- ing. Specifically, our approach uses the Gap Sen- tence Generation objective (GSG) (Zhang et al., …

Content truncated to 500 words