NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji pre-trained CNN for Irony Detection in Tweets

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 638–642 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics NLPRL-IITBHU at SemEval-2018 Task 3: Combining Linguistic Features and Emoji Pre-trained CNN for Irony Detection in Tweets Harsh Rangwani, Devang Kulshreshtha and Anil Kumar Singh Indian Institute of Technology (Banaras Hindu University) Varanasi, India {harsh.rangwani.cse15, devang.kulshreshtha.cse14, aksingh.cse}@iitbhu.ac.in Abstract This paper describes our participation in Se- mEval 2018 Task 3 on Irony Detection in Tweets. We combine linguistic features with pre-trained activations of a neural network. The CNN is trained on the emoji predic- tion task. We combine the two feature sets and feed them into an XGBoost Classifier for classification. Subtask-A involves classifica- tion of tweets into ironic and non-ironic in- stances, whereas Subtask-B involves classifi- cation of tweets into non-ironic, verbal irony, situational irony or other verbal irony. It is observed that combining features from these two different feature spaces improves our sys- tem results. We leverage the SMOTE algo- rithm to handle the problem of class imbal- ance in Subtask-B. Our final model achieves an F1-score of 0.65 and 0.47 on Subtask-A and Subtask-B respectively. Our system ranks 4th on both tasks, respectively, outperforming the baseline by 6% on Subtask-A and 14% on Subtask-B. 1 Introduction According to the Merriam-Webster dictionary1, one of the meanings of irony is defined as ‘the use of words to express something other than and es- pecially the opposite of the literal meaning’ (e.g. I love getting spam emails.). Irony can have dif- ferent forms, such as verbal, situational, dramatic etc. Sarcasm is also categorized as a form of ver- bal irony. Various attempts have been made in the past for detection of sarcasm (Joshi et al., 2017). Sarcastic texts are characterized by the presence of humor and ridicule, which are not always present in the case of ironic texts (Kreuz and Glucksberg, 1989). The absence of these characteristics makes automatic irony detection a more difficult problem than sarcasm detection. 1https://www.merriam-webster.com/ dictionary/irony Irony detection is a problem that is important for the working of many Natural Language Un- derstanding Systems. For example, people often use irony to express their opinions on social media like Twitter (Buschmeier et al., 2014). Detecting irony in social texts can aid in improving opinion analysis. The SemEval 2018 task 3 (Van Hee et al., 2018) consists of two subtasks. Subtask-A involves predicting whether a tweet is ironic or not and Subtask-B involves categorizing a tweet into Non- Ironic, Verbal Irony (by means of a polarity con- trast), Situational Irony and Other Forms of Verbal Irony. The task organizers use macro averaged F1, rather than accuracy to force systems to optimize to work well on all the four classes of tweets, as described in Section 3.1. Systems built in the past primarily used hand- crafted linguistic features for classification of ironic texts (Buschmeier et al. 2014; Far´ıas et al. 2016). In our system, we try to combine them with the pre-trained activations of a neural network. …

Content truncated to 500 words