A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 340–350, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs Kristina Toutanova Microsoft Research Redmond, WA, USA Chris Brockett Microsoft Research Redmond, WA, USA Ke M. Tran∗ University of Amsterdam Amsterdam, The Netherlands Saleema Amershi Microsoft Research Redmond, WA, USA Abstract We introduce a manually-created, multi- reference dataset for abstractive sentence and short paragraph compression. First, we exam- ine the impact of single- and multi-sentence level editing operations on human compres- sion quality as found in this corpus. We ob- serve that substitution and rephrasing opera- tions are more meaning preserving than other operations, and that compressing in context improves quality. Second, we systematically explore the correlations between automatic evaluation metrics and human judgments of meaning preservation and grammaticality in the compression task, and analyze the impact of the linguistic units used and precision ver- sus recall measures on the quality of the met- rics. Multi-reference evaluation metrics are shown to offer significant advantage over sin- gle reference-based metrics. 1 Introduction Automated sentence compression condenses a sen- tence or paragraph to its most important content in order to enhance writing quality, meet document length constraints, and build more accurate docu- ment summarization systems (Berg-Kirkpatrick et al., 2011; Vanderwende et al., 2007). Though word deletion is extensively used (e.g., (Clarke and La- pata, 2008)), state-of-the-art compression models (Cohn and Lapata, 2008; Rush et al., 2015) bene- fit crucially from data that can represent complex abstractive compression operations, including sub- stitution of words and phrases and reordering. ∗This research was conducted during the author’s intern- ship at Microsoft Research. This paper has two parts. In the first half, we in- troduce a manually-created multi-reference dataset for abstractive compression of sentences and short paragraphs, with the following features: • It contains approximately 6,000 source texts with multiple compressions (about 26,000 pairs of source and compressed texts), representing business letters, newswire, journals, and tech- nical documents sampled from the Open Amer- ican National Corpus (OANC1). • Each source text is accompanied by up to five crowd-sourced rewrites constrained to a preset compression ratio and annotated with quality judgments. Multiple rewrites permit study of the impact of operations on human compres- sion quality and facilitate automatic evaluation. • This dataset is the first to provide compressions at the multi-sentence (two-sentence paragraph) level, which may present a stepping stone to whole document summarization. Many of these two-sentence paragraphs are compressed both as paragraphs and separately sentence-by- sentence, offering data that may yield insights into the impact of multi-sentence operations on human compression quality. • A detailed edit history is provided that may allow fine-grained alignment of original and compressed texts and measurement of the cog- nitive load of different rewrite operations. Our analysis of this dataset reveals that abstrac- tion has a significant positive impact on meaning preservation, and that application of trans-sentential 1http://www.anc.org/data/oanc 340 context has a significant …

Content truncated to 500 words