Automatically Assessing Review Helpfulness

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 423–430, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Automatically Assessing Review Helpfulness Soo-Min Kim†, Patrick Pantel†, Tim Chklovski†, Marco Pennacchiotti‡ †Information Sciences Institute University of Southern California 4676 Admiralty Way Marina del Rey, CA 90292 {skim,pantel,timc}@isi.edu ‡ART Group - DISP University of Rome “Tor Vergata” Viale del Politecnico 1 Rome, Italy pennacchiotti@info.uniroma2.it Abstract User-supplied reviews are widely and increasingly used to enhance e- commerce and other websites. Because reviews can be numerous and varying in quality, it is important to assess how helpful each review is. While review helpfulness is currently assessed manu- ally, in this paper we consider the task of automatically assessing it. Experi- ments using SVM regression on a vari- ety of features over Amazon.com product reviews show promising results, with rank correlations of up to 0.66. We found that the most useful features in- clude the length of the review, its uni- grams, and its product rating. 1 Introduction Unbiased user-supplied reviews are solicited ubiquitously by online retailers like Ama- zon.com, Overstock.com, Apple.com and Epin- ions.com, movie sites like imdb.com, traveling sites like citysearch.com, open source software distributors like cpanratings.perl.org, and count- less others. Because reviews can be numerous and varying in quality, it is important to rank them to enhance customer experience. In contrast with ranking search results, assess- ing relevance when ranking reviews is of little importance because reviews are directly associ- ated with the relevant product or service. Instead, a key challenge when ranking reviews is to de- termine which reviews the customers will find helpful. Most websites currently rank reviews by their recency or product rating (e.g., number of stars in Amazon.com reviews). Recently, more sophis- ticated ranking schemes measure reviews by their helpfulness, which is typically estimated by hav- ing users manually assess it. For example, on Amazon.com, an interface allows customers to vote whether a particular review is helpful or not. Unfortunately, newly written reviews and re- views with few votes cannot be ranked as several assessments are required in order to properly es- timate helpfulness. For example, for all MP3 player products on Amazon.com, 38% of the 20,919 reviews received three or fewer helpful- ness votes. Another problem is that low-traffic items may never gather enough votes. Among the MP3 player reviews that were authored at least three months ago on Amazon.com, still only 31% had three or fewer helpfulness votes. It would be useful to assess review helpfulness automatically, as soon as the review is written. This would accelerate determining a review’s ranking and allow a website to provide rapid feedback to review authors. In this paper, we investigate the task of auto- matically predicting review helpfulness using a machine learning approach. Our main contribu- tions are: • A system for automatically ranking reviews according to helpfulness; using state of the art SVM regression, we empirically evaluate our system on a real world dataset collected from Amazon.com on the task of reconstructing …

Content truncated to 500 words