Podlab at SemEval-2019 Task 3: The Importance of Being Shallow

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 292–296 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 292 Podlab at SemEval-2019 Task 3: The Importance of Being Shallow Andrew Nguyen1,2, Tobin South1,2, Nigel G. Bean1,2, Jonathan Tuke1,2 and Lewis Mitchell1,2,3 1ARC Centre of Excellence in Mathematical and Statistical Frontiers 2School of Mathematical Sciences, The University of Adelaide 3Stream Leader, Data to Decisions CRC Abstract This paper describes our linear SVM system for emotion classification from conversational dialogue, entered in SemEval2019 Task 3. We used off-the-shelf tools coupled with feature engineering and parameter tuning to create a simple, interpretable, yet high-performing, classification model. Our system achieves a micro F1 score of 0.7357, which is 92% of the top score for the competition, demonstrating that “shallow” classification approaches can perform well when coupled with detailed fea- ture selection and statistical analysis. 1 Introduction Sentiment analysis from textual data is a well- studied topic, however the particular problem of inferring the emotional context of text-based con- versations is relatively little-studied. With the in- creasing abundance of conversational text-based datasets, particularly from social media, there is increasing interest in studying such datasets across a wide range of domains, from unsolicited opinion polling (Cody et al., 2015) to mental health mon- itoring (Miner et al., 2016). Highly sophisticated systems for emotion detection on conversational data exist (Gupta et al., 2017), however these may not be appropriate for researchers in other fields, and it is desirable to develop interpretable mod- els where the informative features can be readily interpreted and measures of confidence can be as- signed to individual predictions. Our aim in entering SemEval2019 therefore was to build a high-performing model using only off-the-shelf tools and some feature engineering, without resorting to deep neural network archi- tectures or overly sophisticated approaches. This model only uses linear Support Vector Machines (SVMs) (Hastie et al., 2009) and feature engineer- ing using the Natural Language Toolkit (NLTK) (Bird et al., 2009). These relatively “shallow” (as opposed to “deep”) approaches to classification from text have been used, for example, in event prediction from social media data (Tuke et al., 2018). Despite its simplicity, our system per- formed very well, achieving a maximum score in the top 22% of all submissions. 2 Task The task for Semeval 2019 Task 3 is classify- ing the correct emotion (happy, angry, sad or others) on a 3-turn text communication di- alogue (Chatterjee et al., 2019). Table 1 shows an example conversation. Turn 1 You are funny Turn 2 LOL I know that :p Turn 3 =) Label Happy Table 1: A example of the 3-turn text dialogue. The Training dataset contained 5 columns: • ID: a unique number to identify each training sample; • Turn 1: the first turn in the 3-turn conversa- tion, written by User A; • Turn 2: a reply to Turn 1 by User B; • Turn 3: reply to Turn 2, written again by User A; • Label: the human-judged label of …

Content truncated to 500 words