Towards Understanding Omission in Dialogue Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14268–14286 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Understanding Omission in Dialogue Summarization Yicheng Zou1∗, Kaitao Song2†, Xu Tan2, Zhongkai Fu2, Qi Zhang1, Dongsheng Li2, Tao Gui3† 1School of Computer Science, Fudan University, Shanghai, China 2Microsoft Research Asia, China 3Institute of Modern Languages and Linguistics, Fudan University, Shanghai, China {yczou18,qz,tgui}@fudan.edu.cn {kaitaosong,xuta,zhongfu,dongsheng.li}@microsoft.com Abstract Dialogue summarization aims to condense the lengthy dialogue into a concise summary, and has recently achieved significant progress. However, the result of existing methods is still far from satisfactory. Previous works indicated that omission is a major factor in affecting the quality of summarization, but few of them have further explored the omission problem, such as how omission affects summarization results and how to detect omission, which is critical for reducing omission and improving summarization quality. Moreover, analyzing and detecting omission relies on summariza- tion datasets with omission labels (i.e., which dialogue utterances are omitted in the summa- rization), which are not available in the current literature. In this paper, we propose the OLDS dataset, which provides high-quality Omission Labels for Dialogue Summarization. By an- alyzing this dataset, we find that a large im- provement in summarization quality can be achieved by providing ground-truth omission labels for the summarization model to recover omission information, which demonstrates the importance of omission detection for omission mitigation in dialogue summarization. There- fore, we formulate an omission detection task and demonstrate our proposed dataset can sup- port the training and evaluation of this task well. We also call for research action on omission de- tection based on our proposed datasets. Our dataset and codes are publicly available 1. 1 Introduction With the exponential increase in the volume of conversational messages from daily life, there is a growing demand for dialogue summarization (Mur- ray and Carenini, 2008; Gliwa et al., 2019; Chen and Yang, 2020; Zhong et al., 2021; Zou et al., ∗This work was done when the first author was an intern at Microsoft Research Asia. †Corresponding authors. 1https://github.com/microsoft/ MSummarizer/ 2021c), which compresses lengthy interactions into a more concise and structured piece of text while preserving the most important and relevant infor- mation. Recent years have witnessed significant progress in abstractive dialogue summarization, especially using large-scale pre-trained language models (Lewis et al., 2020; Raffel et al., 2020). De- spite the advances in a high level of fluency and coherence, existing models are still prone to gen- erate defective summaries (Kry´sci´nski et al., 2019; Maynez et al., 2020; Tang et al., 2022) that limit their practical usage. Previous works (Chen and Yang, 2020; Liu et al., 2021; Tang et al., 2022) have investigated the taxonomy of errors involved in out- put summaries, and human evaluations revealed that the majority of errors fall into the category of omission, which often leads to incomplete sum- maries where critical facts are lost. However, few of these works have further analyzed the omission problem, let alone addressing this …

Content truncated to 500 words