Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3204 - 3219 July 10-15, 2022 ©2022 Association for Computational Linguistics CURRICULUM: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding Zeming Chen Qiyue Gao Rose-Hulman Institute of Technology {chenz16, gaoq}@rose-hulman.edu Abstract In the age of large transformer language mod- els, linguistic evaluation play an important role in diagnosing models’ abilities and limitations on natural language understanding. However, current evaluation methods show some signif- icant shortcomings. In particular, they do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. Thus they fail to effectively map out the aspects of language understanding that remain challeng- ing to existing models, which makes it hard to discover potential limitations in models and datasets. In this paper, we introduce CUR- RICULUM as a new format of NLI benchmark for evaluation of broad-coverage linguistic phe- nomena. CURRICULUM contains a collection of datasets that covers 36 types of major linguis- tic phenomena and an evaluation procedure for diagnosing how well a language model captures reasoning skills for distinct types of linguis- tic phenomena. We show that this linguistic- phenomena-driven benchmark can serve as an effective tool for diagnosing model behavior and verifying model learning quality. In addi- tion, our experiments provide insight into the limitation of existing benchmark datasets and state-of-the-art models that may encourage fu- ture research on re-designing datasets, model architectures, and learning objectives. 1. 1 Introduction With the rising power of pre-trained language mod- els, large-scale benchmarks serve as an important factor driving the future progress of NLP. These benchmarks can provide a tool for analyzing the strengths and weaknesses of pre-trained language models. In recent years, many benchmarks (Wang et al., 2019, 2020; Rajpurkar et al., 2018) have been proposed that offer a diverse set of evaluation objectives. However, recent criticisms have been 1Our code and data are publicly available at https://github. com/eric11eca/curriculum-ling Figure 1: We propose a broad-coverage diagnostic benchmark for linguistic-phenomena-driven evaluation. Our benchmark includes both a dataset collection and an evaluation procedure for evaluating model performance and diagnosing linguistic skills captured by a model. We evaluate models fine-tuned on large NLI datasets through four types of diagnostic tests: zero-shot, inocu- lation, hypothesis-only, and cross-distribution. made that these benchmarks fail to serve as effec- tive measures of progress in machine learning (Raji et al., 2021). In particular, the task design does not formulate specific linguistic skills required for un- derstanding. They lack the effectiveness in helping researchers understand how certain systems or mod- els work and how they fail. Although many state- of-the-art language models have shown impressive performance on these common benchmarks, their performance degrades considerably on adversarial or out-of-distribution samples (Bras et al., 2020). The performance drop shows that models may not be learning the required linguistic skills for solving the tasks of these benchmarks but exploit spurious dataset biases (Poliak et al., 2018b). Overall, the current …

Content truncated to 500 words