Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11626–11644 July 9-14, 2023 ©2023 Association for Computational Linguistics Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors Liyan Tang♢, Tanya Goyal♢, Alexander R. Fabbri♠, Philippe Laban♠, Jiacheng Xu♢,♠, Semih Yavuz♠, Wojciech Kry´sci´nski♠, Justin F. Rousseau♢, Greg Durrett♢ ♢The University of Texas at Austin ♠Salesforce AI Research lytang@utexas.edu Abstract The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to de- tect factual errors and annotation of errors in current systems’ outputs. However, the ever- evolving nature of summarization systems, met- rics, and annotated benchmarks makes factu- ality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggre- gate factuality error annotations from nine ex- isting datasets and stratify them according to the underlying summarization model. We com- pare performance of state-of-the-art factuality metrics, including recent ChatGPT-based met- rics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Crit- ically, our analysis shows that much of the re- cent improvement in the factuality detection space has been on summaries from older (pre- Transformer) models instead of more relevant recent summarization models. We further per- form a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our re- sults show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.1 1 Introduction Although abstractive summarization systems (Liu and Lapata, 2019; Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020) have improved dra- matically in recent years, these models still of- ten include factual errors in generated summaries (Kryscinski et al., 2020; Maynez et al., 2020). A number of metrics have emerged to detect factuality errors, including methods based on sentence entail- ment (Kryscinski et al., 2020), finer-grained entail- 1Code and data are available at https://github.com/ Liyan06/AggreFact. ment (Goyal and Durrett, 2020; Zhao et al., 2020), question generation and answering (Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021), and discrimination of synthetically-constructed er- ror instances (Cao and Wang, 2021). Despite recent analyses (Pagnoni et al., 2021; Laban et al., 2022), reliably comparing these metrics remains difficult. In this paper, we provide a new benchmark that allows for finer-grained comparison between dif- ferent factuality systems. We aggregate 9 existing annotated factuality datasets to create our bench- mark AGGREFACT. We stratify it according to the underlying summarization model, categorized into FTSOTA, EXFORMER and OLD based on their development timeline (see Section 2). First, we ask: do factuality metrics perform equally well at identifying errors from state-of-the-art sum- marization models and from earlier models? For nine recent factuality metrics, including re- cent ChatGPT-based metrics, we show that metric performance varies substantially between different categories of summarization models. Most impor- tantly, we found that the …

Content truncated to 500 words