Sentiment Analysis: It’s Complicated!

Proceedings of NAACL-HLT 2018, pages 1886–1895 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Sentiment Analysis: It’s Complicated! 1*Kian Kenyon-Dean, 1*Eisha Ahmed, 1†Scott Fujimoto, 1†Jeremy Georges-Filteau, 1†Christopher Glasz, 1†Barleen Kaur, 1†Auguste Lalande, 1#Shruti Bhanderi, 1#Robert Belfer, 1#Nirmal Kanagasabai, 1#Roman Sarrazingendron, 1#Rohit Verma, and 2Derek Ruths 1,2McGill University, Department of Computer Science 1 {first.last}@mail.mcgill.ca 2 derek.ruths@mcgill.ca Abstract Sentiment analysis is used as a proxy to mea- sure human emotion, where the objective is to categorize text according to some prede- fined notion of sentiment. Sentiment anal- ysis datasets are typically constructed with gold-standard sentiment labels, assigned based on the results of manual annotations. When working with such annotations, it is common for dataset constructors to discard “noisy” or “controversial” data where there is significant disagreement on the proper label. In datasets constructed for the purpose of Twitter senti- ment analysis (TSA), these controversial ex- amples can compose over 30% of the origi- nally annotated data. We argue that the re- moval of such data is a problematic trend be- cause, when performing real-time sentiment classification of short-text, an automated sys- tem cannot know a priori which samples would fall into this category of disputed sen- timent. We therefore propose the notion of a “complicated” class of sentiment to categorize such text, and argue that its inclusion in the short-text sentiment analysis framework will improve the quality of automated sentiment analysis systems as they are implemented in real-world settings. We motivate this argu- ment by building and analyzing a new publicly available TSA dataset of over 7,000 tweets annotated with 5x coverage, named MTSA. Our analysis of classifier performance over our dataset offers insights into sentiment analysis dataset and model design, how current tech- niques would perform in the real world, and how researchers should handle difficult data. 1 Introduction The goal of sentiment analysis is to determine the attitude or emotional state held by the author of *These authors contributed equally to this work. †These authors contributed equally to this work. #These authors contributed equally to this work. Tweet text + - 0 Members came in today for lunch to learn more about competitive events. 0 0 5 15 year old with an iPhone X, like DAMN girl, Whatcha gonna do with that much power in your hands? Facebook? Snapchat? That’s it? 0 2 3 i am really missing the food my family makes rn 2 2 1 Table 1: Example tweets from our dataset over varying levels of annotator labellings; +, -, 0 stand for POSI- TIVE, NEGATIVE, OBJECTIVE. a piece of text. Automatic sentiment classifica- tion that can quickly garner user sentiment is use- ful for applications ranging from product market- ing to measuring public opinion. The volume and availability of short-text user content makes auto- mated sentiment analysis systems highly attractive for companies and organizations, despite poten- tial complications arising from their short length and specialized use of language. The popularity of Twitter as a social media platform on which people can readily …

Content truncated to 500 words