Utility is in the Eye of the User: A Critique of NLP Leaderboards

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4846–4853, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4846 Utility is in the Eye of the User: A Critique of NLP Leaderboards Kawin Ethayarajh Stanford University kawin@stanford.edu Dan Jurafsky Stanford University jurafsky@stanford.edu Abstract Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leader- board paradigm has been remarkably success- ful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and en- ergy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in prac- tice through the lens of microeconomic the- ory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards – in their current form – can be poor proxies for the NLP community at large. For exam- ple, a highly inefficient model would provide less utility to practitioners but not to a leader- board, since it is a cost that only the former must bear. To allow practitioners to better es- timate a model’s utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practi- cal concern (e.g., model size, energy efficiency, and inference latency). 1 Introduction The past few years have seen significant progress on a variety of NLP tasks, from question answer- ing to machine translation. These advances have been driven in part by benchmarks such as GLUE (Wang et al., 2018), whose leaderboards rank mod- els by how well they perform on these diverse tasks. Performance-based evaluation on a shared task is not a recent idea either; this sort of shared chal- lenge has been an important driver of progress since MUC (Sundheim, 1995). While this paradigm has been successful at driving the creation of more ac- curate models, the historical focus on performance- based evaluation has been at the expense of other attributes valued by the NLP community, such as fairness and energy efficiency (Bender and Fried- man, 2018; Strubell et al., 2019). For example, a highly inefficient model would have limited use in practical applications, but this would not preclude it from reaching the top of most leaderboards. Sim- ilarly, models can reach the top while containing racial and gender biases – and indeed, some have (Bordia and Bowman, 2019; Manzini et al., 2019; Rudinger et al., 2018; Blodgett et al., 2020). Microeconomics provides a useful lens through which to study the divergence between what is in- centivized by leaderboards and what is valued by practitioners. We can frame both the leaderboard and NLP practitioners as consumers of models and the benefit they receive from a model as its utility to them. Although leaderboards are inanimate, this framing allows us to make an …

Content truncated to 500 words