Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2759–2769 August 1–6, 2021. ©2021 Association for Computational Linguistics 2759 Evaluating Evaluation Measures for Ordinal Classification and Ordinal Quantification Tetsuya Sakai Waseda University / Shinjuku-ku Okubo 3-4-1, Tokyo 169-8555, JAPAN tetsuyasakai@acm.org Abstract Ordinal Classification (OC) is an important classification task where the classes are or- dinal. For example, an OC task for senti- ment analysis could have the following classes: highly positive, positive, neutral, negative, highly negative. Clearly, evaluation measures for an OC task should penalise misclassifica- tions by considering the ordinal nature of the classes (e.g., highly positive misclassified as positive vs. misclassifed as highly negative). Ordinal Quantification (OQ) is a related task where the gold data is a distribution over or- dinal classes, and the system is required to es- timate this distribution. Evaluation measures for an OQ task should also take the ordinal na- ture of the classes into account. However, for both OC and OQ, there are only a small num- ber of known evaluation measures that meet this basic requirement. In the present study, we utilise data from the SemEval and NTCIR communities to clarify the properties of nine evaluation measures in the context of OC tasks, and six measures in the context of OQ tasks. 1 Introduction In NLP and many other experiment-oriented re- search disciplines, researchers rely heavily on eval- uation measures. Whenever we observe an im- provement in the score of our favourite measure, we either assume or hope that this implies that we have managed to moved our system a little towards what we ultimately want to achieve. Hence it is of utmost importance to examine whether evaluation measures are measuring what we want to measure, and to understand their properties. This paper concerns evaluation measures for Or- dinal Classification (OC) and Ordinal Quantifica- tion (OQ) tasks. In an OC task, the classes are ordinal, not nominal. For example, Task 4 (Senti- ment Analysis in Twitter) Subtask C in SemEval- 2016/2017 is defined as: given a set of tweets about a particular topic, estimate the sentiment conveyed by each tweet towards the topic on a five-point scale (highly negative, negative, neutral, positive, highly positive) (Nakov et al., 2016; Rosenthal et al., 2017). On the other hand, an OQ task involves a gold distribution of labels over ordinal classes and the system’s estimated distribution. For exam- ple, Task 4 Subtask E of the SemEval-2016/2017 workshops is defined as: given a set of tweets about a particular topic, estimate the distribution of the tweets across the five ordinal classes already men- tioned above (Nakov et al., 2016; Rosenthal et al., 2017). The Dialogue Breakdown Detection Chal- lenge (Higashinaka et al., 2017) and the Dialogue Quality subtasks of the NTCIR-14 Short Text Con- versation (Zeng et al., 2019) and the NTCIR-15 Dialogue Evaluation (Zeng et al., 2020) tasks are also OQ tasks. 1 Clearly, evaluation measures for OC and OQ tasks …

Content truncated to 500 words