Tübingen-Oslo at SemEval-2018 Task 2: SVMs perform better than RNNs in Emoji Prediction

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 34–38 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Tübingen-Oslo at SemEval-2018 Task 2: SVMs perform better than RNNs at Emoji Prediction Çağrı Çöltekin Department of Linguistics University of Tübingen, Germany ccoltekin@sfs.uni-tuebingen.de Taraka Rama Department of Informatics University of Oslo, Norway tarakark@ifi.uio.no Abstract This paper describes our participation in the SemEval-2018 task Multilingual Emoji Pre- diction. We participated in both English and Spanish subtasks, experimenting with support vector machines (SVMs) and recurrent neural networks. Our SVM classifier obtained the top rank in both subtasks with macro-averaged F1- measures of 35.99 % for English and 22.36 % for Spanish data sets. Similar to a few earlier attempts, the results with neural networks were not on par with linear SVMs. 1 Introduction Emojis are graphical symbols that represent an idea or emotion. The use of emojis has become popular over the last decade, particularly in in- formal communication in the social media. Their popularity kindled a recent interest in investigating many aspects of emojis, including their interaction with natural language (e.g., Barbieri et al., 2016, 2017; Felbo et al., 2017; Kralj Novak et al., 2015). Although the emojis are presumably language- independent, their use typically goes together with linguistic text. In this context, the SemEval 2018 task 2, Multilingual Emoji Prediction (Barbieri et al., 2018), aims predicting the emoji from the surrounding micro-blogging (Twitter) text for En- glish and Spanish. The task at hand is to predict a label, an emoji, from a short text that it accompanies. This is es- sentially a text/document classification problem, and shares many aspects of other text classifica- tion problems such as topic classification, senti- ment analysis, language identification and author- ship attribution – just to name a few. Although each of these problems have some task-specific as- pects, the same models can be used for all of them. In this study, we experiment with and compare two well-known methods: support vector machines (SVMs) with bag of word/character n-gram fea- tures and recurrent neural networks (RNNs) with word and character sequences as input. The meth- ods and implementations are similar to our earlier attempts in other text classification tasks (Çöltekin and Rama, 2016; Rama and Çöltekin, 2017; Çöl- tekin and Rama, 2017).1 In the remainder of this paper, we describe our methods and experiments, present and discuss our results. 2 Experiments and Results We participated in both subtasks using the same architectures. However, we trained and tuned the model parameters on each data set separately. The training set for the competition consisted of 500 000 tweets for English and 100 000 tweets for Spanish subtask. The data sets contained most frequent 20 emojis for English and 19 emojis for Spanish. Joining late to the party, our training set consisted of 485 151 English tweets, and 97 765 Spanish tweets, since about 3 % of the tweets were not available by the time we crawled them. As pre- sented in Figure 1, the …

Content truncated to 500 words