Comparing Test Sets with Item Response Theory

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1141–1158 August 1–6, 2021. ©2021 Association for Computational Linguistics 1141 Comparing Test Sets with Item Response Theory Clara Vania♠∗† Phu Mon Htut♣∗ William Huang♣∗ Dhara Mungra♣ Richard Yuanzhe Pang♣ Jason Phang♣ Haokun Liu♦† Kyunghyun Cho♣ Samuel R. Bowman♣ ♠Amazon ♣New York University ♦Allen Institute for AI vaniclar@amazon.co.uk, bowman@nyu.edu Abstract Recent years have seen numerous NLP datasets introduced to evaluate the perfor- mance of fine-tuned models on natural lan- guage understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely satu- rated and unlikely to be able to detect further progress. What kind of datasets are still ef- fective at discriminating among strong mod- els, and what kind of datasets should we ex- pect to be able to detect future improvements? To measure this uniformly across datasets, we draw on Item Response Theory and eval- uate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, Hel- laSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models, while SNLI, MNLI, and CommitmentBank seem to be saturated for current strong mod- els. We also observe span selection task for- mat, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating be- tween strong and weak models. 1 Introduction Many datasets have been created to evaluate var- ious aspects of natural language understanding (NLU) in English. These datasets are useful to mea- sure progress; however, it is evident from various leaderboards (Wang et al., 2018, 2019b; Rajpurkar et al., 2016; Zellers et al., 2018) that many of them are no longer challenging or discriminative enough to differentiate strong models such as those based on Transformers (Vaswani et al., 2017).1 Even if these benchmarks are sound tests of important ∗Equal contribution. †Work done while at New York University. 1For example, the recent DeBERTa model (He et al., 2020) achieves parity with human annotators on the SuperGLUE benchmark score: https://super.gluebenchmark. com/leaderboard. (and potentially unsolved) tasks, their usefulness is limited if they cannot measure further progress. In this paper, we ask: Which datasets are best in distinguishing current and possible future strong models? We aim to compare datasets using a single metric that accounts for their effectiveness in separating current stronger and weaker models. To that end, we use Item Response Theory (IRT; Baker and Kim, 1993), a statistical framework from psycho- metrics that is widely used for the evaluation of test items in educational assessment. IRT assumes that the probability that a model will correctly handle an example in a test set depends on the model’s latent ability parameter and three example-specific parameters, typically measuring example difficulty (how strong does a model have to be to get it right), discrimination (how effective the example is for differentiating between similar models), and guess- ing (how likely a weak model is to get the example …

Content truncated to 500 words