Duluth UROP at SemEval-2018 Task 2: Multilingual Emoji Prediction with Ensemble Learning and Oversampling

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 482–485 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics Duluth UROP at SemEval-2018 Task 2: Multilingual Emoji Prediction with Ensemble Learning and Oversampling Shuning Jin and Ted Pedersen Department of Computer Science University of Minnesota Duluth, MN 55812 USA {jinxx596,tpederse}@d.umn.edu https://github.com/shuningjin/SemEval2018-Task2-EmojiDetection Abstract This paper describes the Duluth UROP sys- tems that participated in SemEval–2018 Task 2, Multilingual Emoji Prediction. We relied on a variety of ensembles made up of classi- fiers using Naive Bayes, Logistic Regression, and Random Forests. We used unigram and bigram features and tried to offset the skew- ness of the data through the use of oversam- pling. Our task evaluation results place us 19th of 48 systems in the English evaluation, and 5th of 21 in the Spanish. After the evaluation we realized that some simple changes to pre- processing could significantly improve our re- sults. After making these changes we attained results that would have placed us sixth in the English evaluation, and second in the Spanish. 1 Introduction Emoji prediction of tweets is an emerging problem (Barbieri et al., 2017) which combines the nuances of sentiment analysis with the noisy data charac- teristic of social media. SemEval–2018 Task 2 (Barbieri et al., 2018) adds to this the challenge of multilingual processing, where both Spanish and English tweets are used. Given the relatively large amount of training data available for the task (see Section 2) we de- cided to approach this as a classification task, where we used relatively simple unigram and bi- gram features in combination with standard ma- chine learning algorithms. We particularly fo- cused on the use of Multinomial Naive Bayes, Logistic Regression, and Random Forests, all of which were provided to us via the scikit learn package (Pedregosa et al., 2011). Given the chal- lenging nature of this task we decided to employ a variety of ensemble approaches, since no sin- gle classifier seemed likely to perform well in all cases. In the sections that follow we summarize the ex- perimental data used in the task, and then describe the methods we employed for supervised learning, ensemble building, and oversampling. We close by interpreting and discussing our results. 2 Experimental Data The task organizers created both training and test data made up of Spanish and English tweets (sep- arately). The training data consists of 100,000 Spanish tweets and 500,000 English tweets. The test data is made up of 10,000 Spanish tweets and 50,000 English tweets. Each instance consists of a single tweet, where 19 different emojis were ob- served in the Spanish data, and 20 emojis were ob- served in the English. We collected the training tweets via the Twitter API on November 10–11, 2017. By that time some of the tweets selected by the organizers had been deleted or made private, but we were still able to download the vast majority of training tweets. For English we downloaded 490,272 tweets, so 9,628 were unavailable. For Spanish …

Content truncated to 500 words