NELEC at SemEval-2019 Task 3: Think Twice Before Going Deep

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 266–271 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 266 NELEC at SemEval-2019 Task 3: Think Twice Before Going Deep Parag Agrawal∗ Microsoft paragag@microsoft.com Anshuman Suri∗ Microsoft ansuri@microsoft.com Abstract Existing Machine Learning techniques yield close to human performance on text-based classification tasks. However, the presence of multi-modal noise in chat data such as emoti- cons, slang, spelling mistakes, code-mixed data, etc. makes existing deep-learning solu- tions perform poorly. The inability of deep- learning systems to robustly capture these covariates puts a cap on their performance. We propose NELEC : Neural and Lexical Combiner, a system which elegantly com- bines textual and deep-learning based meth- ods for sentiment classification. We eval- uate our system as part of the third task of ’Contextual Emotion Detection in Text’ as part of SemEval-2019 (Chatterjee et al., 2019b). Our system performs significantly better than the baseline, as well as our deep- learning model benchmarks. It achieved a micro-averaged F1 score of 0.7765, ranking 3rd on the test-set leader-board. Our code is available at https://github.com/ iamgroot42/nelec 1 Introduction Sentiment analysis of textual data: Twitter data (Kouloumpis et al., 2011; Pak and Paroubek, 2010), movie reviews (Thet et al., 2010), and prod- uct reviews (Pang et al., 2008), is perhaps the most extensively explored problem, with a plethora of research to tackle it. Novel systems utilise deep learning architectures to achieve near-human per- formance on clean, well-formatted data. How- ever, sentiment classification of chat data is signif- icantly challenging. The presence of spelling er- rors, slang, emoticons, code-mixing, style of writ- ing and abbreviations makes it significantly harder for existing deep-learning models to work on such data. ∗Equal contribution, order determined by coin toss Literature dealing with this problem comprises a wide range of approaches: from hand-crafted features to end-to-end deep-learning methods. Some rule-learning based methods use keyword- based analysis (Ko and Seo, 2000) and part-of- speech tagging (Agarwal et al., 2011). These pro- cedures require extensive human-involvement for identifying keywords and designing rules and are thus not scalable. Non-neural machine-learning methods utilize feature extraction algorithms like n-grams and Tf- Idf vectors, coupled with classification algorithms like Naive Bayes (Pang et al., 2002), Decision Trees (Bilal et al., 2016), SVM (Moraes et al., 2013). These approaches perform significantly better than rule-based approaches but fail to cap- ture context well, since they ignore the order of words in text sequences. Statistic Train Dev Test Emojis (%) 17.6 11.1 12.5 OOV (%) 3.7 4.9 4.9 OOV(processed) (%) 2.1 1.5 1.8 Avg.Length 13.6 12.7 12.7 Avg.Length(processed) 15.7 15.3 15.2 Happy emotion (%) 14.1 5.2 5.2 Sad emotion (%) 18.1 4.5 4.5 Angry emotion (%) 18.3 5.4 5.4 Table 1: Some statistics for the given training, develop- ment and test sets. Neural, deep-learning based approaches use ar- chitectures such as variations of recurrent mod- els: GRU (Chung et al., 2014), LSTM (Hochreiter and Schmidhuber, 1997), BiLSTM (Schuster and Paliwal, 1997) and Convolutional models (Mundra et al., 2017), …

Content truncated to 500 words