Finding a Balanced Degree of Automation for Summary Evaluation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6617–6632 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6617 Finding a Balanced Degree of Automation for Summary Evaluation Shiyue Zhang Mohit Bansal UNC Chapel Hill {shiyue, mbansal}@cs.unc.edu Abstract Human evaluation for summarization tasks is reliable but brings in issues of repro- ducibility and high costs. Automatic met- rics are cheap and reproducible but some- times poorly correlated with human judg- ment. In this work, we propose flexible semi- automatic to automatic summary evaluation metrics, following the Pyramid human evalu- ation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but re- places the manual work of judging SCUs’ presence in system summaries with a natu- ral language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role la- beling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and re- tain SCUs that are more difficult to simulate, which provides a smooth transition and bal- ance between automation and manual evalua- tion. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 ex- isting meta-evaluation datasets and our newly- collected PyrXSum (with 100/10 XSum ex- amples/systems). It shows that Lite2Pyramid consistently has the best summary-level cor- relations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection.1 1 Introduction Evaluating the quality of summaries is a challeng- ing task. Human evaluation is usually regarded as the gold standard. Out of different human evalua- tion methods, Pyramid (Nenkova and Passonneau, 1Our code and data are publicly available at: https: //github.com/ZhangShiyue/Lite2-3Pyramid 2004) has been perceived as an objective and re- liable protocol and used by early summarization benchmarks, e.g., TAC (DBL, 2008, 2009). Given one or several reference summaries of an exam- ple, human assessors first exhaustively extract Sum- mary Content Units (SCUs), each SCU contains a single fact, from the reference(s), and then check whether they are present in a system summary. Fig- ure 1 shows an example of human-labeled SCUs. Despite the reliability, manual evaluation is usually: (1) not reproducible, results may change when dif- ferent evaluators are involved, making it hard to compare the results across papers; (2) expensive, in terms of time and cost. Thus, it is unlikely to apply human evaluation extensively in model selec- tion (e.g., to choose the best checkpoint); instead, people usually treat it as an additional quality verifi- cation step. Aiming to work as a proxy of humans, many automatic metrics have been proposed (Lin, 2004; Tratz and Hovy, 2008; Giannakopoulos and Karkaletsis, 2011; Yang et al., 2016; Zhang et al., 2019; Deutsch et al., 2021). However, most of them cannot reliably substitute human evaluation due to the unstable performance across datasets (Bhandari et al., 2020), weak to moderate correlations with …

Content truncated to 500 words