Evaluating the Efficacy of Summarization Evaluation across Languages

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 801–812 August 1–6, 2021. ©2021 Association for Computational Linguistics 801 Evaluating the Efficacy of Summarization Evaluation across Languages Fajri Koto Jey Han Lau Timothy Baldwin School of Computing and Information Systems The University of Melbourne ffajri@student.unimelb.edu.au, jeyhan.lau@gmail.com, tbaldwin@unimelb.edu.au Abstract While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first at- tempt to systematically quantify their panlin- guistic efficacy. We take a summarization cor- pus for eight different languages, and manu- ally annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English. 1 Introduction Although manual evaluation (Nenkova and Passon- neau, 2004; Hardy et al., 2019) of text summa- rization is more reliable and interpretable, most research on text summarization employs automatic evaluations such as ROUGE (Lin, 2004), ME- TEOR (Lavie and Agarwal, 2007), MoverScore (Zhao et al., 2019), and BERTScore (Zhang et al., 2020b) because they are time- and cost-efficient. In proposing these metrics, the authors measured correlation with human judgments based on En- glish datasets that are not representative of modern summarization systems. For instance, Lin (2004) use DUC1 2001–2003 for ROUGE (meaning sum- maries were generated with largely outdated ex- tractive summarization systems); Zhao et al. (2019) use the TAC2 dataset for MoverScore (again, fea- turing summaries from largely defunct systems; see Peyrard (2019) and Rankel et al. (2013)); and Zhang et al. (2020b) developed BERTScore based on a machine translation corpus (WMT). In contem- poraneous work, Bhandari et al. (2020) address this issue by annotating English CNN/DailyMail sum- maries produced by recent summarization models, and found disparities over results from TAC. 1https://duc.nist.gov/data.html 2https://tac.nist.gov/data/ �Gold�summary�:�Info-A;�Info-B;�Info-C Good�focus,�and�Good�coverage� Good�focus,�and�Bad�coverage��� Bad�focus,�and�Good�coverage Bad�focus,�and�Bad�coverage �:�Info-A;�Info-B;�Info-C� �:�Info-A;�Info-A �:�Info-A;�Info-B;�Info-C;�Info-D;�Info-E �:�Info-D;�Info-E;�Info-F �System�summary: Figure 1: Illustration of focus and coverage. Equally troublingly, ROUGE has become the default summarization evaluation metric for lan- guages other than English (Hu et al., 2015; Scialom et al., 2020; Ladhak et al., 2020; Koto et al., 2020b), despite there being no systematic validation of its efficacy across other languages. The questions we ask in this study, therefore, are twofold: (1) How well do existing automatic metrics perform over languages other than English? and (2) What automatic metric works best across different lan- guages? In this paper, we examine content-based summa- rization evaluation from the aspects of precision and recall, in the form of focus and coverage to compare system-generated summaries to ground- truth summaries (see Figure 1). As advocated by Koto et al. (2020a), focus and coverage are more interpretable and fine grained than the harmonic mean (F1 score) of ROUGE. This is also in line with the review of Hardy et al. (2019) on linguistic properties that have been manually evaluated in re- cent summarization research, who found precision and recall to be commonly used to complement ROUGE F1. While it may seem more natural …

Content truncated to 500 words