GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11444 - 11458 December 7-11, 2022 ©2022 Association for Computational Linguistics GENIE Toward Reproducible and Standardized Human Evaluation for Text Generation Daniel Khashabi3∗ Gabriel Stanovsky1,4 Jonathan Bragg1 Nicholas Lourie5∗ Jungo Kasai2 Yejin Choi1,2 Noah A. Smith1,2 Daniel S. Weld1,2 1Allen Institute for AI 2University of Washington 3Johns Hopkins University 4Hebrew University of Jerusalem 5New York University Abstract While often assumed a gold standard, effec- tive human evaluation of text generation re- mains an important, open area for research. We revisit this problem with a focus on pro- ducing consistent evaluations that are repro- ducible—over time and across different popu- lations. We study this goal in different stages of the human evaluation pipeline. In particu- lar, we consider design choices for the annota- tion interface used to elicit human judgments and their impact on reproducibility. Further- more, we develop an automated mechanism for maintaining annotator quality via a prob- abilistic model that detects and excludes noisy annotators. Putting these lessons together, we introduce GENIE: a system for running standardized human evaluations across differ- ent generation tasks. We instantiate GENIE with datasets representing four core challenges in text generation: machine translation, sum- marization, commonsense reasoning, and ma- chine comprehension. For each task, GENIE offers a leaderboard that automatically crowd- sources annotations for submissions, evaluat- ing them along axes such as correctness, con- ciseness, and fluency. We have made the GE- NIE leaderboards publicly available, and have already ranked 50 submissions from 10 differ- ent research groups.1 We hope GENIE encour- ages further progress toward effective, stan- dardized evaluations for text generation. 1 Introduction While the emergence of powerful language models (Radford et al., 2019; Raffel et al., 2020; Lewis et al., 2020) has made text generation omnipresent, effective evaluation of the resulting systems’ per- formance on open-ended generation tasks remains a challenge. This has motivated adoption of hu- man evaluation in recent works (Celikyilmaz et al., 2020; Fabbri et al., 2021), even though it poses ∗Work done at Allen Institute for AI. 1https://genie.apps.allenai.org Seahawks' defense comes up big late against Patriots news article predicted summary BLEU, BERTScore ROUGE, ... automatic eval. human eval. leaderboards Figure 1: The GENIE architecture for evaluating text generation tasks, with a summarization example. Sim- ilar to automatic leaderboards, model developers sub- mit their predictions (top). GENIE then evaluates with a standard human evaluation as well as with automatic metrics (center). These scores are then used to rank and track systems’ performance across time (bottom). several challenges (Clark et al., 2021; Karpinska et al., 2021). First, the estimates of system perfor- mance are not reproducible—over time and various annotator populations. Additionally, the setups are not standardized. Different works use different annotation interfaces, even those working on the same dataset, despite substantial efforts needed for building an appropriate annotation interface and guidelines to extract quality human annotations and filter out noisy annotators. This work presents an investigation toward reli- ably repeatable and standardized human evaluation. First …

Content truncated to 500 words