An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3938–3949 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3938 An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results Enrique Amigó UNED Madrid, Spain enrique@lsi.uned.es Julio Gonzalo UNED Madrid, Spain julio@lsi.uned.es Stefano Mizzaro University of Udine Udine, Italy mizzaro@uniud.it Jorge Carrillo-de-Albornoz UNED Madrid, Spain jcalbornoz@lsi.uned.es Abstract In Ordinal Classification tasks, items have to be assigned to classes that have a relative order- ing, such as positive, neutral, negative in sen- timent analysis. Remarkably, the most popu- lar evaluation metrics for ordinal classification tasks either ignore relevant information (for in- stance, precision/recall on each of the classes ignores their relative ordering) or assume ad- ditional information (for instance, Mean Aver- age Error assumes absolute distances between classes). In this paper we propose a new met- ric for Ordinal Classification, Closeness Evalu- ation Measure, that is rooted on Measurement Theory and Information Theory. Our theoreti- cal analysis and experimental results over both synthetic data and data from NLP shared tasks indicate that the proposed metric captures qual- ity aspects from different traditional tasks si- multaneously. In addition, it generalizes some popular classification (nominal scale) and er- ror minimization (interval scale) metrics, de- pending on the measurement scale in which it is instantiated. 1 Introduction In Ordinal Classification (OC) tasks, items have to be assigned to classes that have a relative or- dering, such as positive, neutral, negative in sen- timent analysis. It is different from n-ary classifi- cation, because it considers ordinal relationships between classes. It is also different from ranking tasks, which only care about relative ordering be- tween items, because it requires category matching; and it is also different from value prediction, be- cause it does not assume fixed numeric intervals between categories. Most research on Ordinal Classification, how- ever, evaluates systems with metrics designed for those other problems. But classification measures ignore the ordering between classes, ranking met- rics ignore category matching, and value prediction metrics are used by assuming (usually equal) nu- meric intervals between categories. In this paper we propose a metric designed to evaluate Ordinal Classification systems which re- lies on concepts from Measurement Theory and from Information Theory. The key idea is defining a general notion of closeness between item value assignments (system output prediction vs gold stan- dard class) which is instantiated into ordinal scales but can be also be used with nominal or interval scales. Our approach establishes closeness between classes in terms of the distribution of items per class in the gold standard, instead of assuming pre- defined intervals between classes. We provide a formal (Section 4) and empirical (Section 5) com- parison of our metric with previous approaches, and both analytical and empirical evidence indi- cate that our metric suits the problem best than the current most popular choices. 2 State of the Art In this section we first summarize the most popular metrics used in OC evaluation campaigns, and then discuss previous work …

Content truncated to 500 words