Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews Peter D. Turney Institute for Information Technology National Research Council of Canada Ottawa, Ontario, Canada, K1A 0R6 peter.turney@nrc.ca Abstract This paper presents a simple unsupervised learning algorithm for classifying reviews as recommended (thumbs up) or not rec- ommended (thumbs down). The classifi- cation of a review is predicted by the average semantic orientation of the phrases in the review that contain adjec- tives or adverbs. A phrase has a positive semantic orientation when it has good as- sociations (e.g., “subtle nuances”) and a negative semantic orientation when it has bad associations (e.g., “very cavalier”). In this paper, the semantic orientation of a phrase is calculated as the mutual infor- mation between the given phrase and the word “excellent” minus the mutual information between the given phrase and the word “poor”. A review is classified as recommended if the average semantic ori- entation of its phrases is positive. The al- gorithm achieves an average accuracy of 74% when evaluated on 410 reviews from Epinions, sampled from four different domains (reviews of automobiles, banks, movies, and travel destinations). The ac- curacy ranges from 84% for automobile reviews to 66% for movie reviews. 1 Introduction If you are considering a vacation in Akumal, Mex- ico, you might go to a search engine and enter the query “Akumal travel review”. However, in this case, Google1 reports about 5,000 matches. It would be useful to know what fraction of these matches recommend Akumal as a travel destina- tion. With an algorithm for automatically classify- ing a review as “thumbs up” or “thumbs down”, it would be possible for a search engine to report such summary statistics. This is the motivation for the research described here. Other potential appli- cations include recognizing “flames” (abusive newsgroup messages) (Spertus, 1997) and develop- ing new kinds of search tools (Hearst, 1992). In this paper, I present a simple unsupervised learning algorithm for classifying a review as rec- ommended or not recommended. The algorithm takes a written review as input and produces a classification as output. The first step is to use a part-of-speech tagger to identify phrases in the in- put text that contain adjectives or adverbs (Brill, 1994). The second step is to estimate the semantic orientation of each extracted phrase (Hatzivassi- loglou & McKeown, 1997). A phrase has a posi- tive semantic orientation when it has good associations (e.g., “romantic ambience”) and a negative semantic orientation when it has bad as- sociations (e.g., “horrific events”). The third step is to assign the given review to a class, recommended or not recommended, based on the average seman- tic orientation of the phrases extracted from the re- view. If the average is positive, the prediction is that the review recommends the item it discusses. Otherwise, the prediction is that the item is not recommended. The PMI-IR algorithm is employed to estimate the semantic orientation of a phrase (Turney, 2001). PMI-IR uses Pointwise Mutual Information (PMI) and …

Content truncated to 500 words