Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4140–4170 July 9-14, 2023 ©2023 Association for Computational Linguistics Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation Yixin Liu∗1 Alexander R. Fabbri∗2 Pengfei Liu3 Yilun Zhao1 Linyong Nan1 Ruilin Han1 Simeng Han1 Shafiq Joty2 Chien-Sheng Wu2 Caiming Xiong2 Dragomir Radev1 1Yale University, 2Salesforce AI, 3Carnegie Mellon University yixin.liu@yale.edu, afabbri@salesforce.com Abstract Human evaluation is the foundation upon which the evaluation of both summarization systems and automatic metrics rests. How- ever, existing human evaluation studies for sum- marization either exhibit a low inter-annotator agreement or have insufficient scale, and an in-depth analysis of human evaluation is lack- ing. Therefore, we address the shortcomings of existing summarization evaluation along the following axes: (1) We propose a modified sum- marization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units and allows for a high inter- annotator agreement. (2) We curate the Ro- bust Summarization Evaluation (RoSE) bench- mark, a large human evaluation dataset consist- ing of 22,000 summary-level annotations over 28 top-performing systems on three datasets. (3) We conduct a comparative study of four human evaluation protocols, underscoring po- tential confounding factors in evaluation setups. (4) We evaluate 50 automatic metrics and their variants using the collected human annotations across evaluation protocols and demonstrate how our benchmark leads to more statistically stable and significant results. The metrics we benchmarked include recent methods based on large language models (LLMs), GPTScore and G-Eval. Furthermore, our findings have impor- tant implications for evaluating LLMs, as we show that LLMs adjusted by human feedback (e.g., GPT-3.5) may overfit unconstrained hu- man evaluation, which is affected by the anno- tators’ prior, input-agnostic preferences, calling for more robust, targeted evaluation methods. 1 Introduction Human evaluation plays an essential role in both assessing the rapid development of summarization systems in recent years (Lewis et al., 2020a; Zhang et al., 2020a; Brown et al., 2020; Sanh et al., 2022; He et al., 2022) and in assessing the ability of auto- matic metrics to evaluate such systems as a proxy ∗ Equal contribution for manual evaluation (Bhandari et al., 2020; Fabbri et al., 2022a; Gao and Wan, 2022). However, while human evaluation is regarded as the gold standard for evaluating both summarization systems and au- tomatic metrics, as suggested by Clark et al. (2021) an evaluation study does not become “gold” auto- matically without proper practices. For example, achieving a high inter-annotator agreement among annotators can be difficult (Goyal et al., 2022), and there can be a near-zero correlation between the annotations of crowd-workers and expert annota- tors (Fabbri et al., 2022a). Also, a human evalua- tion study without a large enough sample size can fail to find statistically significant results due to insufficient statistical power (Card et al., 2020). Therefore, we believe it is important to ensure that human evaluation can indeed serve as a solid foundation for evaluating summarization systems and automatic metrics. For this, we pro- …

Content truncated to 500 words