Evolutionary Data Measures: Understanding the Difficulty of Text Classification Tasks

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 380–391 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 380 Evolutionary Data Measures: Understanding the Difficulty of Text Classification Tasks Edward Collins Wluper Ltd. London, United Kingdom ed@wluper.com Nikolai Rozanov Wluper Ltd. London, United Kingdom nikolai@wluper.com Bingbing Zhang Wluper Ltd. London, United Kingdom bingbing@wluper.com Abstract Classification tasks are usually analysed and improved through new model architectures or hyperparameter optimisation but the underly- ing properties of datasets are discovered on an ad-hoc basis as errors occur. However, under- standing the properties of the data is crucial in perfecting models. In this paper we anal- yse exactly which characteristics of a dataset best determine how difficult that dataset is for the task of text classification. We then propose an intuitive measure of difficulty for text clas- sification datasets which is simple and fast to calculate. We show that this measure gener- alises to unseen data by comparing it to state- of-the-art datasets and results. This measure can be used to analyse the precise source of errors in a dataset and allows fast estimation of how difficult a dataset is to learn. We searched for this measure by training 12 clas- sical and neural network based models on 78 real-world datasets, then use a genetic algo- rithm to discover the best measure of difficulty. Our difficulty-calculating code1 and datasets2 are publicly available. 1 Introduction If a machine learning (ML) model is trained on a dataset then the same machine learning model on the same dataset but with more granular labels will frequently have lower performance scores than the original model (see results in Zhang et al. (2015); Socher et al. (2013a); Yogatama et al. (2017); Joulin et al. (2016); Xiao and Cho (2016); Con- neau et al. (2017)). Adding more granularity to labels makes the dataset harder to learn - it in- creases the dataset’s difficulty. It is obvious that some datasets are more difficult for learning mod- els than others, but is it possible to quantify this 1https://github.com/Wluper/edm 2http://data.wluper.com difficulty? In order to do so, it would be neces- sary to understand exactly what characteristics of a dataset are good indicators of how well models will perform on it so that these could be combined into a single measure of difficulty. Such a difficulty measure would be useful as an analysis tool and as a performance estimator. As an analysis tool, it would highlight precisely what is causing difficulty in a dataset, reducing the time practitioners need spend analysing their data. As a performance estimator, when practitioners ap- proach new datasets they would be able to use this measure to predict how well models are likely to perform on the dataset. The complexity of datasets for ML has been previously examined (Ho and Basu, 2002; Man- silla and Ho, 2004; Bernad´o-Mansilla and Ho, 2005; Maci et al., 2008), but these works focused on analysing feature space data ∈ IRn. These methods do not easily …

Content truncated to 500 words