QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587 - 2601 July 10-15, 2022 ©2022 Association for Computational Linguistics QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization Alexander R. Fabbri Chien-Sheng Wu Wenhao Liu Caiming Xiong Salesforce AI Research {afabbri, wu.jason, wenhao.liu, cxiong}@salesforce.com Abstract Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of re- search, entailment-based and question answer- ing (QA)-based metrics, and different exper- imental setups often lead to contrasting con- clusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based met- rics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classi- fication, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFACTEVAL, that leads to a 14% average improvement over pre- vious QA-based metrics on the SummaC fac- tual consistency benchmark, and also outper- forms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complemen- tary signals and be combined into a single met- ric for a further performance boost. 1 Introduction Text summarization aims to compress long docu- ment(s) into a short and fluent form that preserves salient information. The field has benefited from the application of pretrained methods (Liu and La- pata, 2019; Lewis et al., 2020; Zhang et al., 2020a). However, state-of-the-art models are not always factually consistent with the source documents they are conditioned on (Maynez et al., 2020; Fabbri et al., 2021). Thus, determining the factual consis- tency of a summary remains an essential task. Recent metrics for summarization factual consis- tency can be broadly split into two categories: 1) Entailment-based metrics that determine whether the content in the summary is entailed by the in- put document (Kryscinski et al., 2020; Koto et al., Document The Knicks beat the Rockets. The fans were excited. Summary The Knicks beat the Bucks. Entailment Matrix Selected Answer [Contra, Neutral, Support] the Bucks � 0.90 0.07 0.03 0.02 0.90 0.08 � Generated Question Who did the Knicks beat? QA Output the Rockets Max Support Score Answer Overlap Score 0.08 0.20 Table 1: Toy example of a factual inconsistency be- tween a summary and a source document. Left: The entailment-based metric computes the level of contradic- tion, neutrality, and support between the summary and each source document sentence. The final factual consis- tency metric is calculated as the maximum support score over all source sentences. Right: The QA-based metric first selects a noun-phrase answer from the summary. A QG model then generates an associated question that a QA model answers based on the source document. The answer overlap score of the QA-based metric measures the semantic overlap between the QA model output and the selected answer as the final metric score. 2020) and 2) QA-based metrics that compute a …

Content truncated to 500 words