Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2930 - 2941 July 10-15, 2022 ©2022 Association for Computational Linguistics Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification Yang Xiao1, Jinlan Fu2∗, See-Kiong Ng2, Pengfei Liu3 1Fudan University, 2National University of Singapore, 3Carnegie Mellon University yangxiaocq12@gmail.com, {jinlan,seekiong}@nus.edu.sg, pliu3@cs.cmu.edu Abstract In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first charac- terizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discrimi- native power. We further, taking the text clas- sification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promis- ingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset dis- crimination over unseen datasets. We released all datasets with features explored in this work on DataLab. 1 1 Introduction In natural language processing (NLP) tasks, there are often datasets that we use as benchmarks against which to evaluate machine learning models, either explicitly defined such as GLUE (Wang et al., 2018) and XTREME (Hu et al., 2020) or implicitly bound to the task (e.g., DPedia (Zhang et al., 2015) has become a default dataset for evaluating of text classification systems). Given this mission, one im- portant feature of a good benchmark dataset is the ability to statistically differentiate diverse systems (Bowman and Dahl, 2021). With large pre-trained models consistently improving state-of-the-art per- formance on NLP tasks (Devlin et al., 2018; Lewis et al., 2019), the performances of many of them have reached a plateau (Zhong et al., 2020; Fu et al., 2020). In other words, it is challenging to discrimi- nate a better model using existing datasets (Wang et al., 2019). In this context, we ask the question: ∗Corresponding author 1https://datalab.nlpedia.ai SST1 CR MR QC IMDB ADE ATISYELPDbpedia 0 2,000 4,000 6,000 8,000 #Samples Citations SST1 CR MR QC IMDB ADE ATIS Yelp Dbpedia ≠1 0 1 2 ⁄sva Figure 1: Illustrate different datasets’ distinguishing ability w.r.t top-scoring systems characterized by our measure log(λsva) on text classification and their corre- sponding citations. are all benchmark’s datasets necessary? We use the text classification task as a case study and try to answer the following two sub-questions: RQ1: How can we quantify the distinguishing ability of benchmark datasets? To answer this question, we first design measures with varying calculation difficulties (§4) to judge datasets’ dis- crimination ability based on top-scoring systems’ performances. By exploring correlations among different measures, we then evaluate how reliable a dataset’s discrimination is when discrimination is calculated solely based on overall results that top-scoring systems have achieved and generalize this measure to other NLP tasks. Fig. …

Content truncated to 500 words