Analyzing and Evaluating Faithfulness in Dialogue Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4897 - 4908 December 7-11, 2022 ©2022 Association for Computational Linguistics Analyzing and Evaluating Faithfulness in Dialogue Summarization Bin Wang†, Chen Zhang†, Yan Zhang†, Yiming Chen†, Haizhou Li♮, †, § †National University of Singapore, Singapore ♮The Chinese University of Hong Kong, Shenzhen, China §Kriston AI, China bwang28c@gmail.com Abstract Dialogue summarization is abstractive in na- ture, making it suffer from factual errors. The factual correctness of summaries has the high- est priority before practical applications. Many efforts have been made to improve faithfulness in text summarization. However, there is a lack of systematic study on dialogue summariza- tion systems. In this work, we first perform the fine-grained human analysis on the faithful- ness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. Furthermore, we present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations. Ex- perimental results show that our evaluation schema is a strong proxy for the factual cor- rectness of summarization models. The human- annotated faithfulness samples and the eval- uation toolkit are released to facilitate future research toward faithful dialogue summariza- tion. Code available: https://github. com/BinWang28/FacEval. 1 Introduction Text summarization aims to condense a document into a short paragraph or a single sentence while conveying the core information (El-Kassas et al., 2021). It can be either extractive or abstractive. Extractive summarization methods identify salient sequence spans from the source document and past- ing them together (Dorr et al., 2003; Kobayashi et al., 2015; Zhong et al., 2020). Abstractive sum- marization methods generate completely new sum- mary in a coherent manner (Paulus et al., 2018; Lewis et al., 2020; Zhang et al., 2020; Liu et al., 2021a; Wang et al., 2022). Previous work discov- ered that abstractive summarization suffers from unfaithful outputs, limiting its applicability in real- world scenarios (Kryscinski et al., 2020; Falke et al., 2019; Zhu et al., 2021a; Ladhak et al., 2022). Dialogue: Freddie: Nanna, are you coming to visit us soon? Winnie: Oh darling, Nanna has broken her leg, you’ll have to visit me instead. Freddie: I forgott. Well come soon. Winnie: Good, ask Mummy and Daddy and they will come when they can. Freddie: Yes love you. Leg better soon? Winnie: Yes, quite soon. Tell mummy to ring me. Bye darling xxxxx Summaries: Human: Winnie has broken her leg and will not visit any time soon. Freddie will ask mummy to call Winnie up. ✓ BART: Nanna has broken her leg, so Freddie will have to visit her instead. Nanna will get better soon. ✓ MV-BART: Nanna has broken her leg and Freddie will have to visit Winnie instead. Mummy and Daddy will come to visit them soon. ✗ Coref-BART: Freddie wants to visit Winnie, but Nanna has broken her leg, so he will have to visit her instead. Mummy and Daddy will come when they can. ✗ CondigSum-BART: Winnie’s Nanna has broken her …

Content truncated to 500 words