Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 310–322 November 10–11, 2021. ©2021 Association for Computational Linguistics 310 Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline Ori Ernst1, Ori Shapira1, Ramakanth Pasunuru2, Michael Lepioshkin1, Jacob Goldberger1, Mohit Bansal2, and Ido Dagan1 1Bar-Ilan University 2UNC Chapel Hill {oriern, obspp18, mishamikel}@gmail.com {ram, mbansal}@cs.unc.edu {jacob.goldberger@, dagan@cs.}biu.ac.il Abstract Aligning sentences in a reference summary with their counterparts in source documents was shown as a useful auxiliary summariza- tion task, notably for generating training data for salience detection. Despite its assessed util- ity, the alignment step was mostly approached with heuristic unsupervised methods, typically ROUGE-based, and was never independently optimized or evaluated. In this paper, we pro- pose establishing summary-source alignment as an explicit task, while introducing two ma- jor novelties: (1) applying it at the more accu- rate proposition span level, and (2) approach- ing it as a supervised classification task. To that end, we created a novel training dataset for proposition-level alignment, derived auto- matically from available summarization evalu- ation data. In addition, we crowdsourced dev and test datasets, enabling model development and proper evaluation. Utilizing these data, we present a supervised proposition alignment baseline model, showing improved alignment- quality over the unsupervised approach. 1 Introduction Text summarization aims to extract the salient in- formation out of a single document or a set of topically-related documents, and to generate a co- herent summary. Inherently, summarization needs to address, either explicitly or implicitly, several embedded subtasks, such as salience detection, re- dundancy removal, and text generation. Attempting to cope with this encompassing chal- lenge, summarization research often involves de- veloping models for specific summarization sub- tasks, utilized either as system components or for auxiliary purposes such as training data creation. In this paper, we draw attention to the particu- lar auxiliary task of summary-source alignment, which was utilized as a supporting data generation step in the summarization literature, for both sin- gle and multi document summarization. Given a gold summarization dataset, the task aligns infor- mation pieces in a reference summary with cor- responding information in the source documents. Such alignments were generated automatically over large summarization datasets, most typically at the full sentence level. Then, (noisy) training data for certain summarization subtasks was automatically derived from these alignments (§2), notably for salience detection (Gehrmann et al., 2018; Chen and Bansal, 2018; Lebanoff et al., 2019), but also for redundancy recognition (Cho et al., 2019) and text rephrasing and fusion (Zhang et al., 2018; Lebanoff et al., 2019). Even though the quality of the subsequent trained models relies on alignment quality, the intermediate alignment methods were neither optimized nor evaluated explicitly, making them difficult to compare and improve. In this paper, we establish summary-source align- ment as a stand-alone auxiliary task, with cor- responding novel datasets, model, and intrinsic evaluation. As a major contribution, aiming to yield more precise alignments, we propose align- ing source-summary information at the more fine- grained level of propositions, rather than at the common sentence …

Content truncated to 500 words