Facet-Aware Evaluation for Extractive Summarization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4941–4957 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4941 Facet-Aware Evaluation for Extractive Summarization Yuning Mao1, Liyuan Liu1, Qi Zhu1, Xiang Ren2, Jiawei Han1 1Department of Computer Science, University of Illinois at Urbana-Champaign, IL, USA 2Department of Computer Science, University of Southern California, CA, USA 1{yuningm2, ll2, qiz3, hanj}@illinois.edu 2xiangren@usc.edu Abstract Commonly adopted metrics for extractive sum- marization focus on lexical overlap at the to- ken level. In this paper, we present a facet- aware evaluation setup for better assessment of the information coverage in extracted sum- maries. Specifically, we treat each sentence in the reference summary as a facet, identify the sentences in the document that express the semantics of each facet as support sen- tences of the facet, and automatically evalu- ate extractive summarization methods by com- paring the indices of extracted sentences and support sentences of all the facets in the refer- ence summary. To facilitate this new evalua- tion setup, we construct an extractive version of the CNN/Daily Mail dataset and perform a thorough quantitative investigation, through which we demonstrate that facet-aware evalu- ation manifests better correlation with human judgment than ROUGE, enables fine-grained evaluation as well as comparative analysis, and reveals valuable insights of state-of-the-art summarization methods.1 1 Introduction Text summarization has enjoyed increasing pop- ularity due to its wide applications, whereas the evaluation of text summarization remains challeng- ing and controversial. The most commonly used evaluation metric of summarization is lexical over- lap, i.e., ROUGE (Lin, 2004), which regards the system and reference summaries as sequences of tokens and measures their n-gram overlap. However, recent studies (Paulus et al., 2017; Schluter, 2017; Kryscinski et al., 2019) reveal the limitations of ROUGE and find that in many cases, it fails to reach consensus with human judgment. Since lexical overlap only captures information 1Data can be found at https://github.com/ morningmoni/FAR. Reference: Three people in Kansas have died from a listeria outbreak. Lexical Overlap: But they did not appear identical to listeria sam- ples taken from patients infected in the Kansas outbreak. (ROUGE-1 F1=37.0, multiple token matches but totally different semantics) Manual Extract: Five people were infected and three died in the past year in Kansas from listeria that might be linked to blue bell creameries products, according to the CDC. (ROUGE-1 F1=36.9, semantics covered but lower ROUGE due to the presence of other details) Reference: Chelsea boss Jose Mourinho and United manager Louis van Gaal are pals. Lexical Overlap: Gary Neville believes Louis van Gaal’s greatest achievement as a football manager is the making of Jose Mourinho. Manual Extract: The duo have been friends since they first worked together at Barcelona in 1997 where they enjoyed a successful relationship at the Camp Nou. (ROUGE Recall/F1=0, no lexical overlap at all) Table 1: Lexical overlap — finding the document sen- tence with the highest ROUGE against one reference sentence — could be misleading. Examples are from the CNN/Daily Mail dataset (Nallapati …

Content truncated to 500 words