Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2109 - 2115 July 10-15, 2022 ©2022 Association for Computational Linguistics Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio Yizhu Liu1, Qi Jia1, Kenny Q. Zhu2∗ Shanghai Jiao Tong University, Shanghai, China 1{liuyizhu, Jia_qi}@sjtu.edu.cn 2kzhu@cs.sjtu.edu.cn Abstract A document can be summarized in a number of ways. Reference-based evaluation of sum- marization has been criticized for its inflexibil- ity. In this paper, we propose a new automatic reference-free evaluation metric that compares semantic distribution between source docu- ment and summary by pretrained language models and considers summary compression ratio. The experiments show that this metric is more consistent with human evaluation in terms of coherence, consistency, relevance, flu- ency. 1 Introduction Summarization evaluation metrics that measure the quality of generated summaries are very impor- tant for the development of summarization systems (Rush et al., 2015; Chopra et al., 2016; Nallapati et al., 2017; Liu et al., 2018, 2022; Lewis et al., 2020; Zhang et al., 2020a; Liu et al., 2021). Most previous summarization evaluation metrics need human-annotated summaries as reference and mea- sure summary quality through the similarity be- tween generated summaries and their reference summaries (Papineni et al., 2002; Lin, 2004; Gane- san, 2006; Ng and Abrecht, 2015; Zhang et al., 2020b; Zhao et al., 2019). Such reference-based evaluation metrics cannot accurately evaluate the summary, because a document has many correct but different summaries. It is difficult and expensive to write many reference summaries by human for eval- uation. Thus, it is useful to develop reference-free evaluation metrics for this task. In this paper, we focus on reference-free eval- uation metrics. As shown in Figure 1, a high- quality summary should be concise and contain the most important information of its document. Some reference-free evaluation metrics (Shao et al., 2017; Gao et al., 2020) unsupervisedly construct a pseudo ∗ The corresponding author. Source Document Mexican restaurant Chipotle has decided to tap into the $70 billion food delivery market by teaming up with an app to bring burritos straight to customers' doors. The fast-casual chain will work with the Postmates app to begin offering delivery for online and mobile orders in 67 cities. But Mexican food fans should know that the restaurant plans to add a nine per cent service charge - with the delivery fees for Postmates beginning at $5 and up, depending on distance and demand. High-quality Summary Chipotle will now be available for delivery with the Postmates app. Online and mobile orders will be available in 67 cities. Low-quality Summary 67 cities will be available for delivery fees for Chipotle. Postmates app orders will be available in Online and mobile. Figure 1: A document with its high-quality and low- quality summaries. The heat map marks the salient con- tent in the document. The darker the colour, the more salient the content. reference summary by selecting salient sentences from the source document, which also ignore the variety of …

Content truncated to 500 words