Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Proceedings of the 43rd Annual Meeting of the ACL, pages 115–124, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales Bo Pang �� � and Lillian Lee �� �� � (1) Department of Computer Science, Cornell University (2) Language Technologies Institute, Carnegie Mellon University (3) Computer Science Department, Carnegie Mellon University Abstract We address the rating-inference problem, wherein rather than simply decide whether a review is “thumbs up” or “thumbs down”, as in previous sentiment analy- sis work, one must determine an author’s evaluation with respect to a multi-point scale (e.g., one to five “stars”). This task represents an interesting twist on stan- dard multi-class text categorization be- cause there are several different degrees of similarity between class labels; for ex- ample, “three stars” is intuitively closer to “four stars” than to “one star”. We first evaluate human performance at the task. Then, we apply a meta- algorithm, based on a metric labeling for- mulation of the problem, that alters a given � -ary classifier’s output in an ex- plicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide signifi- cant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appro- priate to the problem. 1 Introduction There has recently been a dramatic surge of inter- est in sentiment analysis, as more and more people become aware of the scientific challenges posed and the scope of new applications enabled by the pro- cessing of subjective language. (The papers col- lected by Qu, Shanahan, and Wiebe (2004) form a representative sample of research in the area.) Most prior work on the specific problem of categorizing expressly opinionated text has focused on the bi- nary distinction of positive vs. negative (Turney, 2002; Pang, Lee, and Vaithyanathan, 2002; Dave, Lawrence, and Pennock, 2003; Yu and Hatzivas- siloglou, 2003). But it is often helpful to have more information than this binary distinction provides, es- pecially if one is ranking items by recommendation or comparing several reviewers’ opinions: example applications include collaborative filtering and de- ciding which conference submissions to accept. Therefore, in this paper we consider generalizing to finer-grained scales: rather than just determine whether a review is “thumbs up” or not, we attempt to infer the author’s implied numerical rating, such as “three stars” or “four stars”. Note that this differs from identifying opinion strength (Wilson, Wiebe, and Hwa, 2004): rants and raves have the same strength but represent opposite evaluations, and ref- eree forms often allow one to indicate that one is very confident (high strength) that a conference sub- mission is mediocre (middling rating). Also, our task differs from ranking not only because one can be given a single item to classify (as opposed to a set of items to be ordered relative to one another), but because there are settings in which classification is harder than ranking, and vice versa. One can apply …

Content truncated to 500 words