Sentiment Analysis of Reviews: Should we analyze writer intentions or reader perceptions?

Proceedings of Recent Advances in Natural Language Processing, pages 415–419, Hissar, Bulgaria, 7-13 September 2013. Sentiment Analysis of Reviews: Should we analyze writer intentions or reader perceptions? Isa Maks and Piek Vossen Vu University, Faculty of Arts De Boelelaan 1105, 1081 HV Amsterdam e.maks@vu.nl, p.vossen@vu.nl Abstract Many sentiment-analysis methods for the classification of reviews use training and test-data based on star ratings provided by reviewers. However, when reading re- views it appears that the reviewers’ rat- ings do not always give an accurate measure of the sentiment of the review. We performed an annotation study which showed that reader perceptions can also be expressed in ratings in a reliable way and that they are closer to the text than the reviewer ratings. Moreover, we applied two common sentiment-analysis tech- niques and evaluated them on both reader and reviewer ratings. We come to the conclusion that it would be better to train models on reader ratings, rather than on reviewer ratings (as is usually done). 1 Introduction There is a growing volume of product reviews on the web which help customers to make de- cisions when planning to travel or buying a product. Sentiment-analysis tools try to discov- er user opinions in these reviews by converting the text to numerical ratings. Building these tools requires a large set of annotated data to train the classifiers. Most developers compile a training and test corpus by collecting reviews from web sites on which customers post their reviews and give a star rating. They test and train their tools against these reviewer ratings assuming that they are an accurate measure of the sentiment of the review. However, when reading reviews and comparing them with the reviewer ratings there does not always seem to be a clear and con- sistent relation between these ratings and the text (cf. also Carrillo de Albornoz et al., 2011). That is, from a reader’s perspective, there is a discrepancy between what the reviewer ex- presses with the numerical rating and what is expressed in text. For example, the following hotel review was rated ‘7’ (weakly positive), whereas possible guests probably would not go to the hotel after having read the review. The hotel seems rather outdated. The break- fast room is just not big enough to cope with the Sunday-morning crowds. This mismatch between the reviewer’s rating and the review’s sentiment may lead to prob- lems. For example, reviews are often ranked according to their reviewer’s ratings from high- ly positive to highly negative. If the review text is not in accordance with its ranking, the rank- ings may become ineffective. In the area of sentiment analysis and opinion mining the mismatch may lead to methodological prob- lems. Testing and training of sentiment- analysis tools on reviewer ratings may lead to the wrong results if the mismatch between the ratings and the text proves to be a common phenomenon. We assume that one of the most im- portant sources of this mismatch is the fact that the reviewer writes …

Content truncated to 500 words