Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6825–6845 July 9-14, 2023 ©2023 Association for Computational Linguistics Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization Rongxin Zhu Jianzhong Qi Jey Han Lau School of Computing and Information Systems The University of Melbourne rongxinz1@student.unimelb.edu.au, {jianzhong.qi,laujh}@unimelb.edu.au Abstract A series of datasets and models have been proposed for summaries generated for well- formatted documents such as news articles. Di- alogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annota- tions named DIASUMFACT. We define fine- grained factual error detection as a sentence- level multi-label classification problem, and we evaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal re- sults, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candi- date ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue sum- maries, which call for further studies, for which our dataset and results offer a solid foundation.1 1 Introduction Factual inconsistency in abstractive summariza- tion — a phenomenon where model-generated sum- maries contain facts that are inconsistent with the source document — is a widely known problem and has been studied extensively in the document summarization community. An example is shown in Figure 1, where the source document is a di- alogue — the type of documents that this paper focuses on. Existing work covers topics on factual inconsis- tency including error typology and factuality an- notations of state-of-the-art neural summarization models (Maynez et al., 2020; Huang et al., 2020; Pagnoni et al., 2021; Goyal and Durrett, 2021; Fab- bri et al., 2021; Gao and Wan, 2022; Tang et al., 2022a), automatic factual error detectors (Wang et al., 2020; Goyal and Durrett, 2020; Kryscinski 1The dataset and code are available at https://github. com/731935354/Dia-Sum-Fact Lilly: Wanna go out tonight? Marshall: can't :( money's low Lilly: my treat :) Marshall: I wouldn't let a woman pay for me. Source Dialogue Factually Consistent Summary Lilly offered to treat Marshall and he rejected. Factually Inconsistent Summary Lilly offered to treat Marshall and he accepted. Figure 1: Example summaries that are factually consis- tent and inconsistent with a source dialogue. et al., 2020; Durmus et al., 2020; Zeng et al., 2021; Scialom et al., 2021), methods to correct factual errors in summaries (Cao et al., 2020; Dong et al., 2020; Chen et al., 2021a) and methods to produce factually more consistent summaries (Zhao et al., 2020; Cao and Wang, 2021; Tang et al., 2022b; Zhu et al., 2021; Aralikatte et al., 2021; Chen et al., 2021b; Balachandran et al., 2022). Almost all of these works focus on news summarization based on two datasets: CNN/DAILYMAIL (Hermann et al., 2015; Nallapati et al., 2016) and XSUM (Narayan et al., 2018). Dialogue summarization (cf Figure 1), which aims …

Content truncated to 500 words