BrainEE at SemEval-2019 Task 3: Ensembling Linear Classifiers for Emotion Prediction

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 137–141 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 137 BrainEE at SemEval-2019 Task 3: Ensembling Linear Classifiers for Emotion Prediction Vachagan Gratian Universit¨at Stuttgart vgratian@utopianlab.am Abstract We present a homogeneous ensemble of lin- ear perceptrons trained for emotion classifi- cation as part of the SemEval-2019 shared- task 3. The model uses a matrix of proba- bilities to weight the activations of the base- classifiers and makes a final prediction us- ing the sum rule. The base-classifiers are multi-class perceptrons utilizing character and word n-grams, part-of-speech tags and senti- ment polarity scores. The results of our exper- iments indicate that the ensemble outperforms the base-classifiers, but only marginally. In the best scenario our model attains an F-Micro score1 of 0.672, whereas the base-classifiers attained scores ranging from 0.636 to 0.666. 1 Introduction Our task is to detect emotions in multi-turn chat messages (see examples in table 1). The four emotion categories the model has choose from are happy, sad, angry and others. A major caveat of the task is the imbalance of class dis- tribution in the dataset, as described in 4.1. The dataset, as well as the task itself are described in detail in (Chatterjee et al., 2019). We choose to deploy ensemble of linear classi- fiers for this task, rather than a single model for a number of reasons. Firstly, given the inherent ambiguity of emotions (Brainerd, 2018) we expect that ensembles are better suited for any emotion prediction task. Secondly, it has been shown that ensembles are more immune to overfitting in sim- ilar tasks (Dong and Han, 2004). And finally, a single model trained on a large number of feature sets, tend to perform significantly worse than an ensemble where each model is trained on a differ- ent subset (or combinations) of feature types. 1This is a custom F-Micro score. See more details under 4.3 Evaluation Conversation Emotion A: Yes B: How so? A: Don’t message me ever angry A: I am fine B: I am good how is ur week A: I am single others Table 1: Two samples from the dataset with angry and others respectively as gold labels. For this purpose, we deploy BrainT, a multi- class perceptron model utilizing word n-grams and POS-tags, built and trained for implicit emotion detection in Tweets (Gratian and Haid, 2018). In the current scenario, we extend the feature sets of BrainT with character n-grams and Sentiment po- larity scores. We combine n = 11 and n = 5 classifiers into an ensemble model where a fi- nal prediction is made based on the activations. Our model also calculates a matrix of probabili- ties used to weigh the input activations. Each el- ement in the matrix is the probability of a given node making correct prediction for a given emo- tion class. In the initial experiments the nodes are trained on the full train data. In the second group of experiments, nodes are …

Content truncated to 500 words