The Rating Game: Sentiment Rating Reproducibility from Text

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2527–2532, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. The Rating Game: Sentiment Rating Reproducibility from Text Lasse Borgholt, Peter Simonsen, Dirk Hovy Center for Language Technology University of Copenhagen {fkr838|cbl123}@alumni.ku.dk, dirk.hovy@hum.ku.dk Abstract Sentiment analysis models often use rat- ings as labels, assuming that these rat- ings reflect the sentiment of the accom- panying text. We investigate (i) whether human readers can infer ratings from re- view text, (ii) how human performance compares to a regression model, and (iii) whether model performance is affected by the rating “source” (i.e. original author vs. annotator). We collect IMDb movie reviews with author-provided ratings, and have them re-annotated by crowdsourced and trained annotators. Annotators re- produce the original ratings better than a model, but are still far off in more than 5% of the cases. Models trained on annotator-labels outperform those trained on author-labels, questioning the useful- ness of author-rated reviews as training data for sentiment analysis. 1 Introduction Machine learning approaches have become the dominant paradigm for sentiment analysis since introduced by Pang et al. (2002). approaches produce good results,While these they need to be trained on sufficiently large labeled data sets. Since human annotation can be both slow and expensive, many studies use data with an inherent subjectivity indicator, such as movie or product reviews with user ratings (Dave et al., 2003; Pang and Lee, 2005; Snyder and Barzilay, 2007; Elming et al., 2014, i.a.). While it is a fair assumption that the rating expresses the author’s attitude towards the subject, it is less obvious to what extent the review text reflects this attitude, and hence what the relation between text and rating is. In this study, we ask (i) whether readers are able to infer the author’s numerical rating based on the author’s review text, (ii) how well learning algorithms perform on the task compared to human readers, and (iii) whether model performance is affected by the rating source used for labeling (i.e. how the numerical rating is obtained) . In order to investigate these questions, we com- pile a data set of user-generated movie reviews with author ratings and collect both crowdsourced annotator ratings and trained annotator ratings. This setup allows us to evaluate the reproducibility of ratings for both humans and models. We address (i) by comparing author ratings to crowdsourced and trained annotator ratings. Author ratings supposedly capture the essence of the author’s sentiment, but we do not expect annotators to perfectly reproduce these ratings based on text alone. We investigate (ii) by evaluating a linear re- gression model on author-labeled data. Sentiment analysis models supposedly emulate the cognitive process of text-based rating inference. The gap between human and model performance is inter- esting, because if human annotators are unable to consistently infer author ratings, we cannot expect learning algorithms to achieve this goal. Finally, we address (iii) by comparing re- gression models trained on data labeled with crowdsourced and author ratings. …

Content truncated to 500 words