Says Who…? Identification of Expert versus Layman Critics’ Reviews of Documentary Films

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2122–2132, Osaka, Japan, December 11-17 2016. Says Who…? Identification of Expert versus Layman Critics’ Reviews of Documentary Films Ming Jiang School of Information Sciences University of Illinois at Urbana-Champaign {mjiang17@illinois.edu} Jana Diesner School of Information Sciences University of Illinois at Urbana-Champaign {jdiesner@illinois.edu} Abstract We extend classic review mining work by building a binary classifier that predicts whether a review of a documentary film was written by an expert or a layman with 90.70% accuracy (F1 score), and compare the characteristics of the predicted classes. A variety of standard lexical and syntactic features was used for this supervised learning task. Our results suggest that experts write comparatively lengthier and more detailed reviews that feature more complex grammar and a higher diversity in their vocabulary. Layman reviews are more subjective and contextualized in peoples’ everyday lives. Our error analysis shows that laymen are about twice as likely to be mistaken as experts than vice versa. We argue that the type of author might be a useful new feature for improving the accuracy of predicting the rating, helpfulness and authenticity of reviews. Finally, the outcomes of this work might help researchers and practitioners in the field of impact assessment to gain a more fine-grained understanding of the perception of different types of media consumers and reviewers of a topic, genre or information product. 1 Introduction Product reviews help customers to make purchase decisions, and producers to improve and develop goods (Hu & Liu, 2004; Kim et al., 2006; Mudambi & Schuff, 2010). Scalable NLP-based solutions have been developed to support various aspects of these decision-making processes: (1) Describing, understanding and anticipating product ratings can help manufacturers to comprehend a market. Ranking reviews and reviewers further aids this step. The rating values per review are typically user-generated, ordinal variables, often on a 5-point scale (Jiang & Diesner, 2016; Pang & Lee, 2005). (2) Identifying the trustworthiness or authenticity of reviews can assist in separating authentic from fudged reviews (Jindal & Liu, 2007; Jindal et al., 2010; Wu et al., 2010). Predicting this feature is more challenging than the previously mentioned ones as authenticity values are not explicitly provided by reviewers or readers, but need to be inferred from the content of reviews and related metadata. (3) Predicting whether a review was written by an expert or a layman helps to differentiate the impact of the electronic word-of-mouth on the e-marketplace. McAuley and Leskovec (2013) studied the change of reviewers’ expertise over time in order to improve personal recommender systems. Knowing the type of reviewer can also assist with asserting the credibility of reviewers (Basuroy et al., 2003; Flanagin & Metzger, 2013; Liu et al., 2008). Besides commercially motivated analyses of product reviews, assessing the impact of information products such as books, films and other works of art on individuals, groups and society is another domain where knowing the type of author can be beneficial. In the context of impact assessment, …

Content truncated to 500 words