Challenges of Evaluating Sentiment Analysis Tools on Social Media

Challenges of Evaluating Sentiment Analysis Tools on Social Media Diana Maynard, Kalina Bontcheva University of Sheffield, Department of Computer Science Regent Court, 211 Portobello, Sheffield, S1 4DP, UK {d.maynard,k.bontcheva}@sheffield.ac.uk Abstract This paper discusses the challenges in carrying out fair comparative evaluations of sentiment analysis systems. Firstly, these are due to differences in corpus annotation guidelines and sentiment class distribution. Secondly, different systems often make different assumptions about how to interpret certain statements, e.g. tweets with URLs. In order to study the impact of these on evaluation results, this paper focuses on tweet sentiment analysis in particular. One existing and two newly created corpora are used, and the performance of four different sentiment analysis systems is reported; we make our annotated datasets and sentiment analysis applications publicly available. We see considerable variations in results across the different corpora, which calls into question the validity of many existing annotated datasets and evaluations, and we make some observations about both the systems and the datasets as a result. Keywords: evaluation, sentiment analysis, social media, annotation 1. Introduction In the last decade, there has been a plethora of research on various forms of sentiment analysis (Pang and Lee, 2008), from tools to analyse product reviews through to more com- plex tasks such as understanding and predicting political voting from social media. Indeed, the terms opinion mining and sentiment analysis are now used interchangeably, and can be used to cover opinion detection and classification, emotion classification, opinion reliability, opinion stance, strength of opinion, detection of opinion holders and tar- gets, and more. However, fair evaluation of such systems is fraught with difficulty, partly because the task is often sub- jective (human annotators do not agree on what is correct), and partly because comparing systems fairly is complicated when they tackle the problem in different ways. In this pa- per, we discuss some of the typical problems with compar- ing sentiment analysis tools, creating annotated corpora and interpreting the results in a meaningful way, and study the impact of this on 3 different corpora. The DecarboNet project1 aims at understanding the poten- tial of social platforms in mitigating climate change. Within the project, we have developed tools for analysing environ- mental tweets with respect to the opinions expressed and topics discussed, investigating correlations between social media engagement and behavioural change (Maynard and Bontcheva, 2015; Dietzel and Maynard, 2015). We have evaluated the tools in a case study based around the Earth Hour events (Fern´andez et al., 2015), by comparing with state of the art tools on both an existing twitter corpus and a domain-specific crowdsourced evaluation corpus we have created. The datasets are available from our DecarboNet project pages2. These efforts have motivated the discussion in the rest of this paper. 2. Creation of a manually annotated sentiment corpus: Earth Hour 2015 Earth Hour is an annual global event where people switch off their lights for one hour to show they care about the 1http://www.decarbonet.eu/ 2https://gate.ac.uk/projects/decarbonet/ future of the planet. We created a twitter corpus by …

Content truncated to 500 words