A Simple Log-based Loss Function for Ordinal Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 4604–4609 October 12–17, 2022. 4604 A simple log-based loss function for ordinal text classification François Castagnos, Martin Mihelich, Charles Dognin Glanceable {francois, martin, charles}@glanceable.io Abstract The cross-entropy loss function is widely used and generally considered the default loss func- tion for text classification. When it comes to ordinal text classification where there is an ordinal relationship between labels, the cross- entropy is not optimal as it does not incorpo- rate the ordinal character into its feedback. In this paper, we propose a new simple loss func- tion called ordinal log-loss (OLL). We show that this loss function outperforms state-of-the- art previously introduced losses on four bench- mark text classification datasets. 1 Introduction For many classification tasks, there is an order on the labels of the target variable. In particular, in natural language processing (NLP) when, for ex- ample, we are trying to predict the number of stars associated with a review: it is obvious that when the label is 1 star, predicting 2 stars is better than predicting 5 stars. This type of classification is called ordinal classification (or ordinal regression) and many techniques have been developed in recent years around it. Among the most used techniques, the ordinal binary classification consists in de- composing the ordered target variable in several binary ones (Frank and Hall, 2001; Allwein et al., 2000). The threshold methods treat the target vari- able (with N classes) as a continuous real-valued variable and N − 1 thresholds are introduced (Her- brich et al., 2000; Verwaeren et al., 2012; Cao et al., 2020). In the loss-sensitive classification the loss function is built such that a higher penalty is as- signed if the distance between the prediction and the label is higher. Several losses can be used here: mapping the labels {C1; C2; ...; CN} into values {1; 2; ...; N} and use the mean squared error. The margin loss or the hinge loss can also be extended for ordinal regression (Rennie and Srebro, 2005). The weighted kappa loss (de la Torre et al., 2018), the earth mover’s distance (Hou et al., 2016), the soft labels (Diaz and Marathe, 2019; Bertinetto et al., 2020) or the CORAL method (Cao et al., 2020) are other examples of recent modified losses introduced in ordinal classification problems. In order to measure the performance of the ordi- nal regression there are well known metrics such as the off-by-k-accuracy, the mean absolute error, the mean squared error or Kendall Tau for instance (Cardoso and Sousa, 2011; Gaudette and Japkow- icz, 2009). 1.1 Specific contribution The main contribution of this paper is to introduce a new loss named ordinal log-loss (OLL). This loss is easy to use, adapted to ordinal classification and gives more accurate results than classical existing methods in text classification. The idea behind the OLL is to penalize bad predictions instead of rewarding good predictions like the majority of the losses mentioned before do. In section 2 we introduce the …

Content truncated to 500 words