On Extending Neural Networks with Loss Ensembles for Text Classification

On Extending Neural Networks with Loss Ensembles for Text Classification Hamideh Hajiabadi Ferdowsi University of Mashhad (FUM) Mashhad, Iran hamideh.hajiabadi@mail.um.ac.ir Diego Molla-Aliod Macquarie University Sydney, New South Wales, Australia diego.molla-aliod@mq.edu.au Reza Monsefi Ferdowsi University of Mashhad (FUM) Mashhad, Iran monsefi@um.ac.ir Abstract Ensemble techniques are powerful ap- proaches that combine several weak learn- ers to build a stronger one. As a meta learning framework, ensemble techniques can easily be applied to many machine learning techniques. In this paper we propose a neural network extended with an ensemble loss function for text clas- sification. The weight of each weak loss function is tuned within the train- ing phase through the gradient propaga- tion optimization method of the neural net- work. The approach is evaluated on sev- eral text classification datasets. We also evaluate its performance in various en- vironments with several degrees of label noise. Experimental results indicate an improvement of the results and strong re- silience against label noise in comparison with other methods. 1 Introduction In statistics and machine learning, ensemble meth- ods use multiple learning algorithms to obtain better predictive performance (Mannor and Meir, 2001). It has been proved that ensemble meth- ods can boost weak learners whose accuracies are slightly better than random guessing into arbi- trarily accurate strong learners (Bai et al., 2014; Zhang et al., 2016). When it could not be possi- ble to directly design a strong complicated learn- ing system, ensemble methods would be a possible solution. In this paper, we are inspired by ensem- ble techniques to combine several weak loss func- tions in order to design a stronger ensemble loss function for text classification. In this paper we will focus on multi-class clas- sification where the class to predict is encoded as a vector y with the one-hot encoding of the target label, and the output of a classifier ˆy = f(x; ✓) is a vector of probability estimates of each label given input sample x and training parameters ✓. Then, a loss function L(y, ˆy) is a positive function that measures the error of estimation (Steinwart and Christmann, 2008). Different loss functions have different properties, and some well-known loss functions are shown in Table 1. Different loss functions lead to different Optimum Bayes Estimators having their own unique characteris- tics. So, in each environment, picking a specific loss function will affect performance significantly (Xiao et al., 2017; Zhao et al., 2010). In this paper, we propose an approach for com- bining loss functions which performs substantially better especially when facing annotation noise. The framework is designed as an extension to reg- ular neural networks, where the loss function is re- placed with an ensemble of loss functions, and the ensemble weights are learned as part of the gradi- ent propagation process. We implement and eval- uate our proposed algorithm on several text classi- fication datasets. The paper is structured as follows. An overview of several loss functions for classification is briefly introduced in Section 2. The proposed framework and the proposed …

Content truncated to 500 words