Rogue Scores

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1914–1934 July 9-14, 2023 ©2023 Association for Computational Linguistics Rogue Scores Max Grusky max@analyses.org Abstract Correct, comparable, and reproducible model evaluation is essential for progress in machine learning. Over twenty years, thousands of lan- guage and vision models have been evaluated with a popular metric called ROUGE. Does this widespread benchmark metric meet these three evaluation criteria? This systematic review of over two thousand publications using ROUGE finds: (A) Critical evaluation decisions and pa- rameters are routinely omitted, making most re- ported scores irreproducible. (B) Differences in evaluation protocol are common, affect scores, and impact the comparability of results reported in many papers. (C) Thousands of papers use nonstandard evaluation packages with software defects that produce provably incorrect scores. Estimating the overall impact of these findings is difficult: because software citations are rare, it is nearly impossible to distinguish between correct ROUGE scores and incorrect “rogue scores.” 1 1 Introduction This work outlines a major research integrity issue that affects thousands of machine learning papers in dozens of language and vision tasks over a span of nearly twenty years. We discover that the majority of model evaluations using the benchmark ROUGE metric are not reproducible and that ROUGE scores reported in thousands of papers may be incorrect. Evaluation metric integrity is critical for model development and comparison. Researchers evalu- ate models to quantify their behaviors, successes, and failures; to compare new modeling approaches consistently against prior work; and to keep track of progress on challenging tasks. Because sharing code and parameters for models is still uncommon, researchers depend on model evaluation scores re- ported in papers to be comparable and correct. For these reasons, systematic errors in model evaluation may have major consequences for the findings and future trajectory of entire research fields, especially for widely used evaluation metrics like ROUGE. 1Software and data available at: RogueScores.com (A) ROUGE scores are hard to reproduce. Machine learning model evaluations using ROUGE are less reproducible than other scientific fields. 2,834 language model evaluations using ROUGE 20% reproducible 100 psychology studies — Open Sci. Collab. (2015) 39% reproducible 18 economics studies — Camerer et al. (2016) 61% reproducible 21 social science studies — Camerer et al. (2018) 62% reproducible 112 cancer biology studies — Errington et al. (2021) 46% reproducible (B) ROUGE scores are difficult to compare. Model evaluations omit critical details that affect scoring, affecting the comparability of results. Release code — including incomplete and nonfunctional 33% papers Release code with ROUGE evaluation 12% papers Perform ROUGE significance testing / bootstrapping 6% papers List ROUGE configuration parameters 5% papers Cite ROUGE software package — including unofficial 35% papers (C) ROUGE scores are often incorrect. Model evaluations are frequently performed using untested, incorrect ROUGE software packages. Percentage of ROUGE package citations that reference software with scoring errors 76% papers Figure 1: Overview of our systematic review of ROUGE model evaluation. We discover major research integrity issues …

Content truncated to 500 words