Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics

Findings of the Association for Computational Linguistics: ACL 2022, pages 3759 - 3765 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics Daniel Deutsch and Dan Roth Department of Computer and Information Science University of Pennsylvania {ddeutsch,danroth}@seas.upenn.edu Abstract Question answering-based summarization evaluation metrics must automatically deter- mine whether the QA model’s prediction is correct or not, a task known as answer verifi- cation. In this work, we benchmark the lexical answer verification methods which have been used by current QA-based metrics as well as two more sophisticated text comparison methods, BERTScore and LERC. We find that LERC out-performs the other methods in some settings while remaining statistically indistinguishable from lexical overlap others. However, our experiments reveal thatin improved verification performance does not necessarily translate to overall QA-based metric quality: In some scenarios, using a worse verification method — or using none at all — has comparable performance to using the best verification method, a result that we attribute to properties of the datasets.1 1 Introduction A recent trend in summarization metrics is evalu- ating the quality of a summary via question an- swering (QA; Eyal et al., 2019; Scialom et al., 2019, 2021; Durmus et al., 2020; Wang et al., 2020; Deutsch et al., 2021a). These metrics compare the semantic content of two texts (e.g., the reference and candidate summaries) by generating questions from one and answering those questions against the other. The amount of common semantic content is proportional to the number of questions which are answered correctly. A critical step of QA-based evaluation metrics is to verify whether the QA model’s prediction is correct, a task known as answer verification (see Fig. 1). This helps to both suppress noisy output from the QA model as well as identify inconsistent information across the texts. 1Our code is available at http://cogcomp.org/ page/publication_view/966. … He was rescued by his parents before emergency responders arrived on the scene … Source Text BERTScore: 0.20 EM: 0.0 F1: 0.0 LERC: 0.62 … His parents jumped in and pulled him to safety before paramedics arrived … Answer Verification Scores Target Text He was rescued by his parents before what arrived on the scene? Question Figure 1: In the answer verification task, the metrics score how likely two phrases from different contexts have the same meaning. Here, the metrics at the bot- tom score the similarity between “emergency respon- ders,” which was used to generate the question from the source text, and “paramedics,” the predicted answer from a QA model in the target text. Answer verification is typically done by compar- ing the prediction to the expected answer by the exact match or token F1 string comparison methods (Rajpurkar et al., 2016). However, more sophisti- cated text comparison methods have been proposed in recent years, and it is unknown whether they provide a benefit in this particular scenario. In this work, we benchmark various answer verification strategies for QA-based summariza- tion evaluation metrics. Our goal is to …

Content truncated to 500 words