Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 300–309 November 10–11, 2021. ©2021 Association for Computational Linguistics 300 Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries Daniel Deutsch and Dan Roth Department of Computer and Information Science University of Pennsylvania ddeutsch,danroth@seas.upenn.edu Abstract Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a ref- erence. Ideally, this comparison should mea- sure the summary’s information quality by cal- culating how much information the summaries have in common. In this work, we analyze the token alignments used by ROUGE and BERTScore to compare summaries and argue that their scores largely cannot be interpreted as measuring information overlap. Rather, they are better estimates of the extent to which the summaries discuss the same topics. Fur- ther, we provide evidence that this result holds true for many other summarization evalua- tion metrics. The consequence of this result is that the most frequently used summariza- tion evaluation metrics do not align with the community’s research goal, to generate sum- maries with high-quality information. How- ever, we conclude by demonstrating that a re- cently proposed metric, QAEval, which scores summaries using question-answering, appears to better capture information quality than cur- rent evaluations, highlighting a direction for future research.1 1 Introduction The development of a reliable metric that can au- tomatically evaluate the content of a summary has been an active area of research for nearly two decades. Over the years, many different metrics have been proposed (Lin, 2004; Hovy et al., 2006; Giannakopoulos et al., 2008; Louis and Nenkova, 2013; Zhao et al., 2019; Zhang et al., 2020; Deutsch et al., 2021). The majority of these of approaches score a candidate summary by measuring its similarity to a reference summary. The most popular metric, ROUGE (Lin, 2004), compares summaries based 1Our code is available at https://github.com/ CogComp/content-analysis-experiments. Nadal lost to Federer Candidate Summary 1 Federer beat Nadal yesterday Reference Summary Federer and Nadal’s match was close Candidate Summary 2 Different information, correct topic Similar info. Figure 1: Both candidate summaries are similar to the reference, but along different dimensions: Candidate 1 contains some of the same information, whereas candi- date 2’s information is different, but it at least discusses the correct topic. The goal of this work is to understand if summarization evaluation metrics’ scores should be interpreted as measures of information overlap or, less desirably, topic similarity. on their lexical overlap, whereas more recent meth- ods, such as BERTScore (Zhang et al., 2020), do so based on the similarity of the summary tokens’ contextualized word embeddings. Ideally, metrics that evaluate the content of a summary should measure the quality of the infor- mation in the summary. However, it is not clear whether metrics such as ROUGE and BERTScore evaluate summaries based on how much informa- tion they have in common with the reference or some less desirable dimension of similarity, such as whether the two summaries …

Content truncated to 500 words