SNU IDS at SemEval-2019 Task 3: Addressing Training-Test Class Distribution Mismatch in Conversational Classification

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 312–317 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 312 SNU IDS at SemEval-2019 Task 3: Addressing Training-Test Class Distribution Mismatch in Conversational Classification Sanghwan Bae, Jihun Choi and Sang-goo Lee Department of Computer Science and Engineering Seoul National University, Seoul, Korea {sanghwan,jhchoi,sglee}@europa.snu.ac.kr Abstract We present several techniques to tackle the mismatch in class distributions between train- ing and test data in the Contextual Emotion Detection task of SemEval 2019, by extending the existing methods for class imbalance prob- lem. Reducing the distance between the dis- tribution of prediction and ground truth, they consistently show positive effects on the per- formance. Also we propose a novel neural architecture which utilizes representation of overall context as well as of each utterance. The combination of the methods and the mod- els achieved micro F1 score of about 0.766 on the final evaluation. 1 Introduction A new task whose goal is to predict the emo- tion of the last speaker given a sequence of text messages has been designed, and coined Contex- tual Emotion Detection (EmoContext; Chatterjee et al., 2019). Though predicting the emotion of a single utter- ance or sentence, i.e. emotion detection, is a well- discussed subject in natural language understand- ing literature, EmoContext has several novel chal- lenges. Firstly, the class distribution of training data is significantly different from that of the test data. Consequently, a model trained on the train- ing data might not perform well on the test data. There have been efforts made to address the prob- lem of learning from training data sets that have imbalanced class distribution, i.e. the class imbal- ance problem (Chawla et al., 2004; Buda et al., 2018). However, they are not applicable to our case, since the imbalance appears only in the test data while the training data can be viewed to be balanced. We extend the existing methods of addressing the class imbalance problem to be applicable for more general cases where the distributions of the collected training data differ from those of the real population or the data at test time, under the as- sumption that the validation set is organized care- fully to reflect the practical distribution. From ex- periments and analyses, we show that the proposed methods reduce the difference between two distri- butions and as a result improve the performance of the model. The additional challenge we have to consider arises from the fact that utterances having identical surface form may have different meanings due to sarcasm, irony, or etc.. Thus a model should track the emotional transitions within a dialogue. To grasp the context of utterances, we propose a semi- hierarchical encoder structure. Lastly, the texts contain lots of non-standard words, e.g. emoticons and emojis. This makes it difficult to exploit typ- ical pre-trained embeddings such as GloVe (Pen- nington et al., 2014). Therefore, we adopt pre- trained embeddings which are specialized for han- dling non-standard words. We show that the …

Content truncated to 500 words