HighRES: Highlight-based Reference-less Evaluation of Summarization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3381–3392 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3381 HIGHRES: Highlight-based Reference-less Evaluation of Summarization Hardy1 Shashi Narayan2∗ Andreas Vlachos1,3 1Department of Computer Science, University of Sheffield 2Google Research 3Department of Computer Science and Technology, University of Cambridge hhardy2@sheffield.ac.uk, shashinarayan@google.com, av308@cam.ac.uk Abstract There has been substantial progress in sum- marization research enabled by the availabil- ity of novel, often large-scale, datasets and recent advances on neural network-based ap- proaches. However, manual evaluation of the system generated summaries is inconsistent due to the difficulty the task poses to hu- man non-expert readers. To address this is- sue, we propose a novel approach for manual evaluation, HIGHlight-based Reference-less Evaluation of Summarization (HIGHRES), in which summaries are assessed by multiple an- notators against the source document via man- ually highlighted salient content in the latter. Thus summary assessment on the source doc- ument by human judges is facilitated, while the highlights can be used for evaluating mul- tiple systems. To validate our approach we employ crowd-workers to augment with high- lights a recently proposed dataset and com- pare two state-of-the-art systems. We demon- strate that HIGHRES improves inter-annotator agreement in comparison to using the source document directly, while they help emphasize differences among systems that would be ig- nored under other evaluation approaches.1 1 Introduction Research in automatic summarization has made headway over the years with single document summarization as the front-runner due to the avail- ability of large datasets (Sandhaus, 2008; Her- mann et al., 2015; Narayan et al., 2018b) which has enabled the development of novel methods, many of them employing recent advances in neural networks (See et al., 2017; Narayan et al., 2018c; Pasunuru and Bansal, 2018, inter alia). ∗The work was primarily done while Shashi was still at School of Informatics, University of Edinburgh. 1Our dataset and code are available at https:// github.com/sheffieldnlp/highres Figure 1: Highlight-based evaluation of a summary. Annotators to evaluate a summary (bottom) against the highlighted source document (top) presented with a heat map marking the salient content in the document; the darker the colour, the more annotators deemed the highlighted text salient. Measuring progress in summarization is diffi- cult, as the task has as input a source document consisting of multiple sentences and methods need to generate a shorter text that expresses the salient information of the source fluently and succinctly. Thus there can be multiple equally good sum- maries for the same source document as not all salient information can fit in a given summary length, while even extractive methods that select complete sentences are not guaranteed to produce a coherent summary overall. The most consistently used evaluation approach is comparison of the summaries produces against reference summaries via automatic measures such as ROUGE (Lin, 2004) and its variants. However, 3382 automatic measures are unlikely to be sufficient to measure performance in summarization (Schluter, 2017), also known for other tasks in which the goal is …

Content truncated to 500 words