DialSummEval: Revisiting Summarization Evaluation for Dialogues

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5693 - 5709 July 10-15, 2022 ©2022 Association for Computational Linguistics DialSummEval: Revisiting Summarization Evaluation for Dialogues Mingqi Gao, Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University {gaomingqi,wanxiaojun}@pku.edu.cn Abstract Dialogue summarization is receiving increas- ing attention from researchers due to its ex- traordinary difficulty and unique application value. We observe that current dialogue sum- marization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. In our paper, we re-evaluate 18 categories of metrics in terms of four dimen- sions: coherence, consistency, fluency and rel- evance, as well as a unified human evaluation of various models in dialogue summarization for the first time. Some noteworthy trends which are different from the conventional sum- marization tasks are identified. We will release DialSummEval, a multi-faceted dataset of hu- man judgments containing the outputs of 14 models on SAMSum.1 1 Introduction Neural network based approaches and sizable datasets have led to significant progress in researches towards conventional summarization tasks such as news and scientific papers (Lin and Ng, 2019). Compared with conventional sum- marization tasks, dialogue summarization has re- ceived increasing attention from researchers due to its great difficulty and unique application value (Feng et al., 2021a). With the proposal of dialogue summary datasets such as SAMSum (Gliwa et al., 2019), DialogSum (Chen et al., 2021) and Medi- aSum (Zhu et al., 2021), a number of models for automatic generation of dialogue summaries have emerged (Feng et al., 2021b; Liu and Chen, 2021; Zou et al., 2021; Qi et al., 2021; Chen and Yang, 2020; Chen and Yang, 2021; Zhao et al., 2020; Liu et al., 2021). There is no denying that these studies have made promising progress, but it remains a chal- lenge to evaluate these advances comprehen- sively. Current studies generally use the SAMSum 1Code and data will be available at https://github. com/kite99520/DialSummEval dataset and adopt ROUGE (Lin, 2004), an n-gram- based automatic evaluation metric using reference summaries, as the overall evaluation criterion for summary quality, complemented by manual eval- uation. Schluter (2017) and Graham (2015) illus- trate the limitations of ROUGE in evaluating sum- marization tasks. Also the manual evaluation pro- tocols vary from one research to another based on our observations. We argue that the inadequate evaluation mech- anism may have become a major obstacle to the progress of dialogue summarization researches. Many studies, such as Chen and Yang (2020) and Tang et al. (2021), have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, in- correct reasoning and improper gender pronouns, and ROUGE may not reflect these problems ef- fectively. For example, Gabriel et al. (2021) note that ROUGE-1 and ROUGE-L fail to ac- curately measure factual inconsistency across do- mains. Our case study in Table 1 also illustrates this point. However, it is impractical …

Content truncated to 500 words