Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary

Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary Daniel Deutsch,† Tania Bedrax-Weiss,‡ and Dan Roth† †Department of Computer and Information Science, University of Pennsylvania, United States ‡Google Research, United States {ddeutsch,danroth}@seas.upenn.edu tbedrax@google.com Abstract A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via em- beddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary’s in- formation overlap with a reference, making them fundamentally different than text over- lap metrics. We demonstrate the experimen- tal benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each compo- nent of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1 1 Introduction Evaluating the content quality of a summary is a fundamental task of text summarization. As such, it has received the attention of researchers for the past two decades (Lin, 2004; Nenkova and Passonneau, 2004; Hovy et al., 2006; Louis and 1Code is available at https://github.com/CogComp /qaeval-experiments. Nenkova 2013; Zhao et al., 2019, among others). The most popular approaches are reference-based metrics, which treat a human-written reference summary as the gold standard and score a candi- date summary based on how similar its content is to the reference. It is desirable to have reference-based evalu- ation metrics that calculate this similarity score based on how much information the two summa- ries have in common. The vast majority of previous automatic evaluation metrics compare two summaries based on matching their tokens, either through some lexical (Lin, 2004; Hovy et al., 2006; Tratz and Hovy, 2008) or embedding-based similarity (Zhang et al., 2020; Zhao et al., 2019). Although they capture a valuable quality signal, these methods match tokens that do not express the same information and instead end up comparing the similarity of two summaries based on the topics they discuss (Deutsch and Roth, 2020). In this work, we propose a metric to evaluate the content quality of a summary using question- answering (QA). Metrics within a QA evaluation framework represent the information of a refer- ence summary using QA pairs, then estimate how much of this information is contained in a can- didate summary by calculating the proportion of questions it can answer. Because the questions can only be answered if the candidate summary con- tains the corresponding information, QA-based metrics directly measure the information overlap, providing a summary quality signal that is not effectively captured by text overlap based metrics. We …

Content truncated to 500 words