Sentiment Summarization: Evaluating and Learning User Preferences

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 514–522, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Sentiment Summarization: Evaluating and Learning User Preferences Kevin Lerman Columbia University New York, NY klerman@cs.columbia.edu Sasha Blair-Goldensohn Google, Inc. New York, NY sasha@google.com Ryan McDonald Google, Inc. New York, NY ryanmcd@google.com Abstract We present the results of a large-scale, end-to-end human evaluation of various sentiment summarization models. The evaluation shows that users have a strong preference for summarizers that model sentiment over non-sentiment baselines, but have no broad overall preference be- tween any of the sentiment-based models. However, an analysis of the human judg- ments suggests that there are identifiable situations where one summarizer is gener- ally preferred over the others. We exploit this fact to build a new summarizer by training a ranking SVM model over the set of human preference judgments that were collected during the evaluation, which re- sults in a 30% relative reduction in error over the previous best summarizer. 1 Introduction The growth of the Internet as a commerce medium, and particularly the Web 2.0 phe- nomenon of user-generated content, have resulted in the proliferation of massive numbers of product, service and merchant reviews. While this means that users have plenty of information on which to base their purchasing decisions, in practice this is often too much information for a user to absorb. To alleviate this information overload, research on systems that automatically aggregate and summa- rize opinions have been gaining interest (Hu and Liu, 2004a; Hu and Liu, 2004b; Gamon et al., 2005; Popescu and Etzioni, 2005; Carenini et al., 2005; Carenini et al., 2006; Zhuang et al., 2006; Blair-Goldensohn et al., 2008). Evaluating these systems has been a challenge, however, due to the number of human judgments required to draw meaningful conclusions. Of- ten systems are evaluated piecemeal, selecting pieces that can be evaluated easily and automati- cally (Blair-Goldensohn et al., 2008). While this technique produces meaningful evaluations of the selected components, other components remain untested, and the overall effectiveness of the entire system as a whole remains unknown. When sys- tems are evaluated end-to-end by human judges, the studies are often small, consisting of only a handful of judges and data points (Carenini et al., 2006). Furthermore, automated summariza- tion metrics like ROUGE (Lin and Hovy, 2003) are non-trivial to adapt to this domain as they re- quire human curated outputs. We present the results of a large-scale, end-to- end human evaluation of three sentiment summa- rization models applied to user reviews of con- sumer products. The evaluation shows that there is no significant difference in rater preference be- tween any of the sentiment summarizers, but that raters do prefer sentiment summarizers over non- sentiment baselines. This indicates that even sim- ple sentiment summarizers provide users utility. An analysis of the rater judgments also indicates that there are identifiable situations where one sen- timent summarizer is generally preferred over the others. We attempt to learn …

Content truncated to 500 words