AgreeSum: Agreement-Oriented Multi-Document Summarization

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3377–3391 August 1–6, 2021. ©2021 Association for Computational Linguistics 3377 AgreeSum: Agreement-Oriented Multi-Document Summarization Richard Yuanzhe Pang1∗§ Adam D. Lelkes2∗ Vinh Q. Tran2∗ Cong Yu2 1New York University, New York, NY 10011, USA 2Google Research, New York, NY 10011, USA yzpang@nyu.edu, {lelkes,vqtran,congyu}@google.com Abstract We aim to renew interest in a particular multi- document summarization (MDS) task which we call AgreeSum: agreement-oriented multi- document summarization. Given a cluster of articles, the goal is to provide abstractive summaries that represent information common and faithful to all input articles. Given the lack of existing datasets, we create a dataset for AgreeSum, and provide annotations on article-summary entailment relations for a sub- set of the clusters in the dataset. We aim to create strong baselines for the task by ap- plying the top-performing pretrained single- document summarization model PEGASUS onto AgreeSum, leveraging both annotated clusters by supervised losses, and unannotated clusters by T5-based entailment-related and language-related losses. Compared to other baselines, both automatic evaluation and hu- man evaluation show better article-summary and cluster-summary entailment in generated summaries. On a separate note, we hope that our article-summary entailment annota- tions contribute to the community’s effort in improving abstractive summarization faithful- ness. 1 Introduction Recent works have made great progress in single- document summarization (SDS) thanks to the encoder-decoder framework and pretraining proce- dures (Cho et al., 2014; Rush et al., 2015; Narayan et al., 2018; Zhang et al., 2020a). There is a growing interest in multi-document summariza- tion (MDS; Zopf, 2018; Fabbri et al., 2019; Liu and Lapata, 2019; Chu and Liu, 2019; Wang et al., 2020b, et seq.), with applications in search engines, news clustering, timeline generation, and other ar- eas. Past MDS research has primarily focused on § Work completed during internship at Google Research. ∗ Equal contribution. summarizing articles such that the summary cov- ers an event “comprehensively” while “avoiding redundancy” (Fabbri et al., 2019). We can say that most existing MDS tasks summarize the “union” of the articles. In this paper, we discuss agreement-oriented multi-document summarization (AgreeSum), in which we aim to abstractively summarize the “in- tersection” of the articles. More specifically, the input to the task is a cluster of articles, and the expected output is a summary that represents in- formation common and faithful to all input articles in the cluster (Section 3). A few works (discussed in Section 2) have investigated the problem, with- out using modern neural-network-based methods. The motivation for reviving interest in AgreeSum is twofold. First, given that certain microscopic details are not likely present in all articles in a given cluster, they would be filtered out through AgreeSum. If the source articles reflect different points of view, AgreeSum provides a way of cap- turing the common ground. The second motivation for AgreeSum lies in the pursuit of summarization faithfulness. AgreeSum is timely, given that recent works have shown diffi- culty of producing faithful abstractive summaries (Falke et al., 2019; Maynez et al., 2020; Kryscinski …

Content truncated to 500 words