NTUA-SLP at SemEval-2018 Task 2: Predicting Emojis using RNNs with Context-aware Attention

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 438–444 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics NTUA-SLP at SemEval-2018 Task 2: Predicting Emojis using RNNs with Context-aware Attention Christos Baziotis1,3, Nikos Athanasiou1 Georgios Paraskevopoulos1,4, Nikolaos Ellinas1 Athanasia Kolovou1,2, Alexandros Potamianos1,4 1School of ECE, National Technical University of Athens, Athens, Greece 2 Department of Informatics, University of Athens, Athens, Greece 3 Department of Informatics, Athens University of Economics and Business, Athens, Greece 4 Behavioral Signal Technologies, Los Angeles, CA cbaziotis@mail.ntua.gr, el12074@central.ntua.gr geopar@central.ntua.gr, nellinas@central.ntua.gr akolovou@di.uoa.gr, potam@central.ntua.gr Abstract In this paper we present a deep-learning model that competed at SemEval-2018 Task 2 “Mul- tilingual Emoji Prediction”. We participated in subtask A, in which we are called to pre- dict the most likely associated emoji in En- glish tweets. The proposed architecture relies on a Long Short-Term Memory network, aug- mented with an attention mechanism, that con- ditions the weight of each word, on a “con- text vector” which is taken as the aggregation of a tweet’s meaning. Moreover, we initial- ize the embedding layer of our model, with word2vec word embeddings, pretrained on a dataset of 550 million English tweets. Finally, our model does not rely on hand-crafted fea- tures or lexicons and is trained end-to-end with back-propagation. We ranked 2nd out of 48 teams. 1 Introduction Emojis play an important role in textual commu- nication, as they function as a substitute for non- verbal cues, that are taken for granted in face-to- face communication, thus allowing users to con- vey emotions by means other than words. Despite their large appeal in text, they haven’t received much attention until recently. Former works, mostly consider their semantics (Aoki and Uchida, 2011; Espinosa-Anke et al., 2016; Barbieri et al., 2016b,a; Ljubeši´c and Fišer, 2016; Eisner et al., 2016) and only recently their role in social media was explored (Barbieri et al., 2017; Cappallo et al., 2018). In SemEval-2018 Task 2: “Multilingual Emoji Prediction” (Barbieri et al., 2018), given a tweet, we are asked to predict its most likely asso- ciated emoji. thanks for always making me feel like family ! love you guys ! … Label: Top 5: i better get a bunch of fire type pokemon Label: Top 5: Figure 1: Attention heat-map visualization. The color intensity corresponds to the weight given to each word by the self-attention mechanism. In this work, we present a near state of the art approach for predicting emojis in tweets, which outperforms the best present work (Barbieri et al., 2017). For this purpose, we employ an LSTM network augmented with a context-aware self- attention mechanism, producing a feature repre- sentation used for classification. Moreover, the at- tention mechanism helps us make our model’s be- havior more interpretable, by examining the distri- bution of the attention weights for a given tweet. To this end, we provide visualizations with the dis- tributions of the attention weights. 2 Overview Figure 3 provides a high-level overview of our ap- proach that consists of three main …

Content truncated to 500 words