FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055–5070 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5055 FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization Esin Durmus∗ Cornell University ed459@cornell.edu He He New York University hehe@cs.nyu.edu Mona Diab The George Washington University mtdiab@gwu.edu Abstract Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Ex- isting automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated sum- mary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faith- fulness: outputs with less word overlap with the source document are more likely to be un- faithful. Next, we propose an automatic ques- tion answering (QA) based metric for faithful- ness, FEQA,1 which leverages recent advances in reading comprehension. Given question- answer pairs generated from the summary, a QA model extracts answers from the docu- ment; non-matched answers indicate unfaith- ful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understand- ing models, our QA-based metric has signif- icantly higher correlation with human faithful- ness scores, especially on highly abstractive summaries. 1 Introduction Abstractive summarization models must aggre- gate salient content from the source document(s) and remain faithful, i.e. being factually consis- tent with information in the source documents. Neural abstractive models are effective at identi- fying salient content and producing fluent sum- maries (See et al., 2017; Chen and Bansal, 2018; Gehrmann et al., 2018). However, the generated summary may not always contain faithful infor- mation, which is vital for real-world applications. ∗Most of the work is done while the authors were at Amazon Web Services AI. 1Faithfulness Evaluation with Question Answering. Source. The world’s oldest person has died a few weeks after celebrating her 117th birth- day. Born on March 5, 1898, the great- grandmother had lived through two world wars, the invention of the television and the first successful powered aeroplane flight by the wright brothers... Output sentence. The world ’s oldest person has died on March 5, 1898. Table 1: An example of unfaithful output (highlighted in red); generated by Gehrmann et al. (2018). Table 1 shows an example of unfaithful gener- ation. Recent studies have shown that around 30% of generated summaries contain unfaithful information (Cao et al., 2018; Falke et al., 2019a; Kry´sci´nski et al., 2019), especially when the sen- tence combines content from multiple source sen- tences (Lebanoff et al., 2019). In this paper, we address the problem of eval- uating faithfulness of generated summaries given their source documents. Our key insight is that current models are limited by a trade-off between abstractiveness and faithfulness (Section 2). On a wide range of systems and two datasets with varying levels of abstractiveness (CNN/DM and XSum), we show that the number of unfaithful sentences …

Content truncated to 500 words