Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4486–4503 August 1–6, 2021. ©2021 Association for Computational Linguistics 4486 Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards? Pedro Rodriguez University of Maryland Facebook Reality Labs∗ me@pedro.ai Joe Barrow University of Maryland jdbarrow@cs.umd.edu Alexander Hoyle University of Maryland hoyle@umd.edu John P. Lalor University of Notre Dame john.lalor@nd.edu Robin Jia University of Southern California robinjia@usc.edu Jordan Boyd-Graber University of Maryland jbg@umiacs.umd.edu Abstract Leaderboards are widely used in NLP and push the field forward. While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models). Rather than replace leaderboards, we advocate a re-imagining so that they better highlight if and where progress is made. Building on educa- tional testing, we create a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses. Using this model, we analyze the ranking reliability of leaderboards. Afterwards, we show the model can guide what to annotate, identify annotation errors, detect overfitting, and identify informa- tive examples. We conclude with recommenda- tions for future benchmark tasks. 1 Leaderboards are Shiny Leaderboard evaluations—for better or worse—are the de facto standard for measuring progress in question answering (Rajpurkar et al., 2016) and in many NLP tasks (Wang et al., 2019a). An un- fortunate side effect of leaderboard popularity is SOTA-chasing, often at the expense of carefully inspecting data and models (Linzen, 2020). For example, the same “super-human” models that top question answering leaderboards (Najberg, 2018) often fail spectacularly (Feng et al., 2018; Wallace et al., 2019a) by learning non-generalizable statisti- cal patterns (McCoy et al., 2019; Niven and Kao, 2019). Finally, focusing solely on metrics conflates progress on a specific task with progress on real- world NLP problems behind the task (Bender and Koller, 2020). Plainly, focusing on headline SOTA numbers “provide(s) limited value for scientific progress absent insight into what drives them” and where they fail (Lipton and Steinhardt, 2019). ∗Work completed at University of Maryland. −7.5 −5.5 −3.5 −1.5 0.5 2.5 4.5 6.5 8.5 0 2,000 −8 −6 −4 −2 0 2 4 6 8 10 Difficulty (�) −10 −5 0 5 10 Discriminability (�) 0 2,000 −10.0 −8.0 −6.0 −4.0 −2.0 0.0 2.0 4.0 6.0 8.0 10.0 12.0 0.0 1.0 Feasibility (λ) Annotation Error Discriminative and Hard Discriminative Easy Figure 1: Difficulty and Ability Discriminating (DAD) leaderboards infer the difficulty, discriminativeness, and feasibility of examples. Negative discriminability sug- gests an annotation error; for example, the question with most negative discriminability asks “Why did demand for rentals decrease?” when the answer is “demand for higher quality housing increased.” In this work we take leaderboards “as they are,” and imagine how they might better support re- search. Leaderboards establish differences between models on a fixed task. Hence, leaderboards should enable and encourage the comparison of models and inspection of examples. And leaderboards should also signal when …

Content truncated to 500 words