Dynabench: Rethinking Benchmarking in NLP

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124 June 6–11, 2021. ©2021 Association for Computational Linguistics 4110 Dynabench: Rethinking Benchmarking in NLP Douwe Kiela†, Max Bartolo‡, Yixin Nie⋆, Divyansh Kaushik§, Atticus Geiger¶, Zhengxuan Wu¶, Bertie Vidgen∥, Grusha Prasad⋆⋆, Amanpreet Singh†, Pratik Ringshia†, Zhiyi Ma†, Tristan Thrush†, Sebastian Riedel†‡, Zeerak Waseem††, Pontus Stenetorp‡, Robin Jia†, Mohit Bansal⋆, Christopher Potts¶ and Adina Williams† † Facebook AI Research; ‡ UCL; ⋆ UNC Chapel Hill; § CMU; ¶ Stanford University ∥ Alan Turing Institute; ⋆⋆ JHU; †† Simon Fraser University dynabench@fb.com Abstract We introduce Dynabench, an open-source plat- form for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in- the-loop dataset creation: annotators seek to create examples that a target model will mis- classify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contempo- rary models quickly achieve outstanding per- formance on benchmark tasks but nonethe- less fail on simple challenge examples and falter in real-world scenarios. With Dyn- abench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and infor- mative benchmarks. We report on four ini- tial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic bench- marking as a new standard for the field. 1 Introduction While it used to take decades for machine learning models to surpass estimates of human performance on benchmark tasks, that milestone is now rou- tinely reached within just a few years for newer datasets (see Figure 1). As with the rest of AI, NLP has advanced rapidly thanks to improvements in computational power, as well as algorithmic break- throughs, ranging from attention mechanisms (Bah- danau et al., 2014; Luong et al., 2015), to Trans- formers (Vaswani et al., 2017), to pre-trained lan- guage models (Howard and Ruder, 2018; Devlin et al., 2019; Liu et al., 2019b; Radford et al., 2019; Brown et al., 2020). Equally important has been the rise of benchmarks that support the development of ambitious new data-driven models and that encour- age apples-to-apples model comparisons. Bench- marks provide a north star goal for researchers, and 2000 2005 2010 2015 2020 1.0 0.8 0.6 0.4 0.2 0.0 0.2 MNIST GLUE ImageNet SQuAD 1.1 SQuAD 2.0 Switchboard Figure 1: Benchmark saturation over time for popular benchmarks, normalized with initial performance at mi- nus one and human performance at zero. are part of the reason we can confidently say we have made great strides in our field. In light of these developments, one might be forgiven for thinking that NLP has created mod- els with human-like language capabilities. Prac- titioners know that, despite our progress, we are actually far from this goal. Models that achieve super-human performance on benchmark tasks (ac- cording to the narrow criteria used to define hu- man performance) nonetheless fail on simple chal- lenge examples …

Content truncated to 500 words