Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4566–4575 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4566 Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark Nikita Nangia1 nikitanangia@nyu.edu Samuel R. Bowman1,2,3 bowman@nyu.edu 1Center for Data Science New York University 2Dept. of Linguistics New York University 3Dept. of Computer Science New York University Abstract The GLUE benchmark (Wang et al., 2019b) is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at launch to 83.9, state of the art at the time of writing (May 24, 2019). Here, we measure hu- man performance on the benchmark, in order to learn whether significant headroom remains for further progress. We provide a conser- vative estimate of human performance on the benchmark through crowdsourcing: Our an- notators are non-experts who must learn each task from a brief set of instructions and 20 ex- amples. In spite of limited training, these an- notators robustly outperform the state of the art on six of the nine GLUE tasks and achieve an average score of 87.1. Given the fast pace of progress however, the headroom we ob- serve is quite limited. To reproduce the data- poor setting that our annotators must learn in, we also train the BERT model (Devlin et al., 2019) in limited-data regimes, and conclude that low-resource sentence classification re- mains a challenge for modern neural network approaches to text understanding. 1 Introduction This past year has seen tremendous progress in building general purpose models that can learn good language representations across a range of tasks and domains (McCann et al., 2017; Peters et al., 2018; Devlin et al., 2019; Howard and Ruder, 2018; Liu et al., 2019). Reusable models like these can be readily adapted to different lan- guage understanding tasks and genres. The Gen- eral Language Understanding Evaluation (GLUE; Wang et al., 2019b) benchmark is designed to evaluate such models. GLUE is built around nine sentence-level natural language understand- ing (NLU) tasks and datasets, including instances of natural language inference, sentiment analysis, acceptability judgment, sentence similarity, and common sense reasoning. The recent BigBird model (Liu et al., 2019) —a fine-tuned variant of the BERT model (De- vlin et al., 2019)—is state-of-the-art on GLUE at the time of writing, with the original BERT right at its heels. Both models perform impressively enough on GLUE to prompt some increasingly urgent questions: How much better are humans at these NLP tasks? Do standard benchmarks have enough headroom to meaningfully measure further progress? In the case of one prominent language understanding task with a known hu- man performance number, SQuAD 2.0 (Rajpurkar et al., 2018), models built on BERT come ex- tremely close to human performance.1 On the recent Situations With Adversarial Generations (SWAG; Zellers et al., 2018) dataset, BERT out- performs individual expert human annotators. In this work, we estimate human performance on the GLUE test set …

Content truncated to 500 words