What Will it Take to Fix Benchmarking in Natural Language Understanding?

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855 June 6–11, 2021. ©2021 Association for Computational Linguistics 4843 What Will it Take to Fix Benchmarking in Natural Language Understanding? Samuel R. Bowman New York University bowman@nyu.edu George E. Dahl Google Research, Brain Team gdahl@google.com Abstract Evaluation for many natural language under- standing (NLU) tasks is broken: Unreliable and biased systems score so highly on stan- dard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will per- form poorly, but ultimately only obscures the abilities that we want our benchmarks to mea- sure. In this position paper, we lay out four cri- teria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require sig- nificant progress in the design of benchmark datasets, the reliability with which they are an- notated, their size, and the ways they handle social bias. 1 Introduction A large and impactful thread of research on nat- ural language understanding (NLU) has focused on improving results on benchmark datasets that feature roughly independent and identically dis- tributed (IID) training, validation, and testing sec- tions, drawn from data that were collected or anno- tated by crowdsourcing (Maas et al., 2011; Bow- man et al., 2015; Rajpurkar et al., 2016; Wang et al., 2019b). Recent methodological progress combined with longstanding issues in crowdsourced data qual- ity has made it so state-of-the-art systems are near- ing the maximum achievable values on most of these benchmarks and thus are unlikely to be able to measure further improvements (Devlin et al., 2019; Raffel et al., 2020). At the same time, these apparently high-performing systems have serious known issues and have not achieved human-level competence at their tasks (Ribeiro et al., 2020). 1. Good performance on the benchmark should imply robust in-domain performance on the task. �→ We need more work on dataset design and data collection methods. 2. Benchmark examples should be accurately and unam- biguously annotated. �→ Test examples should be validated thoroughly enough to remove erroneous examples and to prop- erly handle ambiguous ones. 3. Benchmarks should offer adequate statistical power. �→ Benchmark datasets need to be much harder and/or much larger. 4. Benchmarks should reveal plausibly harmful social biases in systems, and should not incentivize the cre- ation of biased systems. �→ We need to better encourage the development and use auxiliary bias evaluation metrics. Figure 1: A summary of the criteria we propose. Progress suffers in the absence of a trustworthy metric for benchmark-driven work: Newcomers and non-specialists are discouraged from trying to contribute, and specialists are given significant freedom to cherry-pick ad-hoc evaluation settings that mask a lack of progress (Church and …

Content truncated to 500 words