GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 353 GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Wang1, Amanpreet Singh1, Julian Michael2, Felix Hill3, Omer Levy2, and Samuel R. Bowman1 1New York University, New York, NY 2Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA 3DeepMind, London, UK {alexwang,amanpreet,bowman}@nyu.edu {julianjm,omerlevy}@cs.washington.edu felixhill@google.com Human ability to understand language is gen- eral, flexible, and robust. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understand- ing beyond the detection of superficial correspon- dences between inputs and outputs, then it is crit- ical to develop a unified model that can execute a range of linguistic tasks across different domains. To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE, gluebenchmark.com): a benchmark of nine diverse NLU tasks, an auxil- iary dataset for probing models for understand- ing of specific linguistic phenomena, and an on- line platform for evaluating and comparing mod- els. For some benchmark tasks, training data is plentiful, but for others it is limited or does not match the genre of the test set. GLUE thus favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. While none of the datasets in GLUE were created from scratch for the benchmark, four of them feature privately-held test data, which is used to ensure that the benchmark is used fairly. We evaluate baselines that use ELMo (Peters et al., 2018), a powerful transfer learning tech- nique, as well as state-of-the-art sentence repre- sentation models. The best models still achieve fairly low absolute scores. Analysis with our diag- nostic dataset yields similarly weak performance over all phenomena tested, with some exceptions. The GLUE benchmark GLUE consists of nine English sentence understanding tasks covering a broad range of domains, data quantities, and diffi- culties. As the goal of GLUE is to spur develop- ment of generalizable NLU systems, we design the benchmark such that good performance should re- Corpus |Train| Task Domain Single-Sentence Tasks CoLA 8.5k acceptability misc. SST-2 67k sentiment movie reviews Similarity and Paraphrase Tasks MRPC 3.7k paraphrase news STS-B 7k textual sim. misc. QQP 364k paraphrase online QA Inference Tasks MNLI 393k NLI misc. QNLI 108k QA/NLI Wikipedia RTE 2.5k NLI misc. WNLI 634 coref./NLI fiction books Table 1: Task descriptions and statistics. Bold de- notes tasks for which there is privately-held test data. All tasks are binary classification, except STS-B (regression) and MNLI (three classes). quire models to share substantial knowledge (e.g., trained parameters) across tasks, while maintain- ing some task-specific components. Though it is possible to train a model per task and evaluate the resulting set of models on this benchmark, we ex- pect that inclusion of several data-scarce tasks will ultimately render …

Content truncated to 500 words