ILDAE: Instance-Level Difficulty Analysis of Evaluation Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3412 - 3425 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ILDAE: Instance-Level Difficulty Analysis of Evaluation Data Neeraj Varshney, Swaroop Mishra, Chitta Baral Arizona State University {nvarshn2, srmishr1, cbaral}@asu.edu Abstract Knowledge of difficulty level of questions helps a teacher in several ways, such as estimating students’ potential quickly by asking carefully selected questions and improving quality of ex- amination by modifying trivial and hard ques- tions. Can we extract such benefits of instance difficulty in Natural Language Processing? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large- scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient- yet-accurate evaluations with fewer instances saving computational cost and time, 2) improv- ing quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) se- lecting the best model based on application re- quirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Com- prehensive experiments for these applications lead to several interesting results, such as eval- uation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall cor- relation with evaluation using complete dataset and computing weighted accuracy using diffi- culty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores1 and hope our work will encourage research in this important yet under- studied field of leveraging instance difficulty in evaluations. 1 Introduction Transformer-based language models (Devlin et al., 2019; Liu et al., 2019; Clark et al., 2020) have im- proved state-of-the-art performance on numerous natural language processing benchmarks (Wang et al., 2018, 2019; Talmor et al., 2019); however, recent studies (Zhong et al., 2021; Sagawa et al., 2020) have raised questions regarding whether these models are uniformly better across all in- stances. This has drawn attention towards instance- 1https://github.com/nrjvarshney/ILDAE Figure 1: Illustrating five applications of Instance-Level Difficulty Analysis of Evaluation data (ILDAE). level analysis of evaluation data (Rodriguez et al., 2021; Vania et al., 2021; Mishra and Arunkumar, 2021) which was previously limited to training data (Swayamdipta et al., 2020; Xu et al., 2020; Mishra and Sachdeva, 2020). Furthermore, it is intuitive that not all instances in a dataset are equally dif- ficult. However, instance-level difficulty analysis of evaluation data (ILDAE) has remained underex- plored in many different ways: what are the poten- tial applications and broad impact associated with ILDAE? In this work, we address the above question by first computing difficulty scores of evaluation in- stances (section 2) and then demonstrating five novel applications of ILDAE (Figure 1). 1. Efficient Evaluations: We propose an ap- proach of conducting efficient-yet-accurate eval- uations. Our approach uses as little as 5% eval- uation instances (selected via ILDAE) to achieve up to 0.93 Kendall correlation with evaluations conducted using the complete dataset. Thus, without considerably impacting the effective- ness of evaluations, our approach saves compu- tational cost and time. 3412 2. Improving …

Content truncated to 500 words