QuestEval: Summarization Asks for Fact-based Evaluation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6594 QuestEval: Summarization Asks for Fact-based Evaluation Thomas Scialom⋆‡, Paul-Alexis Dray⋆, Patrick Gallinari‡, Sylvain Lamprier‡, Benjamin Piwowarski⋄‡, Jacopo Staiano⋆, Alex Wang† ⋄ CNRS, France ‡ Sorbonne Université, CNRS, LIP6, F-75005 Paris, France ⋆ reciTAL, Paris, France † New York University thomas@recital.ai Abstract Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to corre- late poorly with human judgments. To allevi- ate this issue, recent work has proposed evalua- tion metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source doc- ument. Though promising, the proposed ap- proaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QU E S TEVAL. In contrast to established metrics such as ROUGE or BERTScore, QU E S TEVAL does not require any ground- truth reference. Nonetheless, QUESTEV A L substantially improves the correlation with hu- man judgments over four evaluation dimen- sions (consistency, coherence, fluency, and rel- evance), as shown in extensive experiments. We make code and models available.1 1 Introduction The reliability of automatic evaluation metrics is an important factor for progress in artificial intelli- gence tasks, enabling the comparison and improve- ment of the proposed systems. The design of reli- able metrics for natural language generation (NLG) systems is very challenging, and still an open re- search problem: Novikova et al. (2017) and Peyrard (2019) showed that current metrics do not correlate well with human judgments, and argued for the development of new evaluation metrics. Among NLG tasks, summarization is one of the most difficult to evaluate automatically. For a given document, the number of possible correct outputs is much larger than for other tasks such as machine translation. Thus, when only a single reference summary is given – as is typically the case for 1https://github.com/ThomasScialom/QuestEval large-scale summarization datasets, the correlation of standard automatic evaluation metrics with hu- man judgments is low (Louis and Nenkova, 2013). Furthermore, since a summary must be shorter than the corresponding source document, information selection (Li et al., 2018) is critical so that the summary only contains the salient contents from its source document. For these reasons, n-gram based metrics, such as ROUGE (Lin, 2004), are known to poorly reflect human preference (Louis and Nenkova, 2013; Novikova et al., 2017; Paulus et al., 2017; Bhandari et al., 2020). Finally, it is cru- cial for an effective summarization to generate texts that are factually consistent with their source doc- uments. However, this aspect is not measured by n-grams based metrics. Notably, while recent state- of-the-art generative models (Lewis et al., 2019; Zhang et al., 2019a) produce fluent summaries, they frequently contain false or unsupported in- formation (Kry´sci´nski et al., 2019), a phenomenon also known as neural hallucination (Rohrbach et al., 2018; …

Content truncated to 500 words