An Anchor-Based Automatic Evaluation Metric for Document Summarization

Proceedings of the 28th International Conference on Computational Linguistics, pages 5696–5701 Barcelona, Spain (Online), December 8-13, 2020 5696 An Anchor-Based Automatic Evaluation Metric for Document Summarization Kexiang Wang1*, Tianyu Liu1*, Baobao Chang1,2 and Zhifang Sui1,2 1Key Laboratory of Computational Linguistics, Ministry of Education, School of Electronics Engineering and Computer Science, Peking University, Beijing, China 2Peng Cheng Laboratory, Guangdong, China {wkx,tianyu0421,chbb,szf}@pku.edu.cn Abstract The widespread adoption of reference-based automatic evaluation metrics such as ROUGE has promoted the development of document summarization. In this paper, we consider a new pro- tocol for designing reference-based metrics that require the endorsement of source document(s). Following protocol, we propose an anchored ROUGE metric fixing each summary particle on source document, which bases the computation on more solid ground. Empirical results on benchmark datasets validate that source document helps to induce a higher correlation with hu- man judgments for ROUGE metric. Being self-explanatory and easy-to-implement, the protocol can naturally foster various effective designs of reference-based metrics besides the anchored ROUGE introduced here. 1 Introduction Automatic evaluation metric plays a vital role in evaluating system performance for the task of document summarization. Challenges remain in the design of an ideal evaluation metric and the off-the-shelf met- rics have their own drawbacks (Schluter, 2017; Kryscinski et al., 2019). The widely adopted metrics, e.g. ROUGE (Lin, 2004), are reference-based in that they compare the output of some summarizer (namely peer summary) with one or multiple human-authored summaries (namely reference/model summary). The reference-free metrics are still not mature enough to be utilized for evaluation in a real-world setting since their correlations with human judgments have been reported to fall far behind reference-based met- rics, especially for multi-document summarization (Peyrard et al., 2017; Gao et al., 2020)1. In this paper, we consider a new protocol of reference-based summarization metrics by rethinking the role of source document which is indeed a lost treasure neglected by most previous works. Furthermore, a specific implementation of the protocol (i.e. anchored version of ROUGE) will be discussed. The reference-based metrics that already exist typically pursue a kinda computation of overlap be- tween the peer and reference summary either at a lexical level (Lin, 2004) or at a semantic level (Ng and Abrecht, 2015; Sun and Nenkova, 2019; Zhang et al., 2019). However, to our knowledge, few of them consider the impact of source document (or documents in multi-document summarization) on the computation. This goes against common sense as source document is the true information source of both summaries and can be utilized to boost the discriminative power of metrics. Therefore, we advance a new protocol of reference-based metrics for the evaluation of document summarization. More specifi- cally, the direct participation of source document is a necessity to compute any reference-based metric for document summarization. This makes source document endorse a certain metric and the advantage lies with the ability to fact-check the information of peer summary based on the information pool (i.e. source document). The protocol change is illustrated in Figure 1. Metrics designed under the new …

Content truncated to 500 words