Tweety at SemEval-2018 Task 2: Predicting Emojis using Hierarchical Attention Neural Networks and Support Vector Machine

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 497–501 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Tweety at SemEval-2018 Task 2: Predicting Emojis using Hierarchical Attention Neural Networks and Support Vector Machine Daniel Kopev, Atanas Atanasov, Dimitrina Zlatkova, Momchil Hardalov, Ivan Koychev FMI, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria {dkopev, amitkov, dvzlatkova}@uni-sofia.bg {hardalov, koychev}@fmi.uni-sofia.bg Ivelina Nikolova, Galia Angelova IICT, Bulgarian Academy of Sciences, Sofia, Bulgaria {iva, galia}@lml.bas.bg Abstract We present the system built for SemEval- 2018 Task 2 on Emoji Prediction. Although Twitter messages are very short we managed to design a wide variety of features: tex- tual, semantic, sentiment, emotion-, and color- related ones. We investigated different meth- ods of text preprocessing including replacing text emojis with respective tokens and splitting hashtags to capture more meaning. To rep- resent text we used word n-grams and word embeddings. We experimented with a wide range of classifiers and our best results were achieved using a SVM-based classifier and a Hierarchical Attention Neural Network. 1 Introduction SemEval 2018 Task 2 on Emoji Prediction (Bar- bieri et al., 2018) is a classical task for supervised learning. Given labeled data consisting of Twitter messages and a corresponding emoji as a label, the aims is to classify new examples (tweets) into 20 categories - the most frequent emojis of two lan- guages: English (Subtask 1) and Spanish (Subtask 2). We participated only in Subtask 1. The labels are presented in Figure 1: Figure 1: Labels ordered by frequency. 2 Related Work Prior work includes using LSTM-RNN and CNN models (Zhao and Zeng) utilizing pre-trained Twitter embeddings with the latter achieving very good results. Other works (Barbieri et al., 2017) show that LSTMs have high accuracy and even outperform humans at the emoji prediction task. In (Barbieri et al., 2016) the skip-gram neural em- bedding model is applied with different dimen- sions of the vectors and length of the windows ap- plied to both words and emojis. 3 Data We used the 500k training and 50k trial tweets provided by the organizers to train and validate our models respectively. One key mistake we made is that we did not compare those two datasets for duplicate entries. As we found out only after the submission deadline, the train and trial data had a 40% overlap, which unfortunately skewed our ex- pected results and made them unrealistically high. The experimental results presented in Table 2 are on the data with removed duplicates. We crawled additional 100k tweets via Tweepy1 only 5k of which were compliant with the require- ments to contain exactly one emoji. With this ex- ternal data we aimed to improve the overall per- formance of our models, but since it was way too small, it did not have much effect. Finally, when predicting on the test data, we trained our models on the combined train, trial and crawled data. Looking at the emojis we immediately noticed two problematic groups: 1. two emojis with a camera - one …

Content truncated to 500 words