Assessing Objective Recommendation Quality through Political Forecasting

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2348–2357 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Assessing Objective Recommendation Quality through Political Forecasting H. Andrew Schwartz† Masoud Rouhizadeh†,‡ Michael Bishop‡ Philip Tetlock‡ Barbara Mellers‡ Lyle H. Ungar⊙ †Computer Science, Stony Brook University ‡Psychology, University of Pennsylvania ⊙Computer & Information Science, University of Pennsylvania has@cs.stonybrook.edu, ungar@cis.upenn.edu Abstract Recommendations are often rated for their subjective quality, but few researchers have studied quality in terms of objec- tive utility. We explore quality assess- ment with respect to both subjective (i.e. users’ ratings) and objective (i.e., did it in- fluence? did it improve decisions?) met- rics in a massive online geopolitical fore- casting system, ultimately comparing lin- guistic characteristics of each quality met- ric. Using a variety of features, we predict all types of quality with better accuracy than the simple yet strong baseline of rec- ommendation length. For example, more complex sentence constructions, as evi- denced by subordinate conjunctions, are characteristic of recommendations leading to objective improvements in forecasting. Our analyses also reveal rater biases; for example, forecasters are subjectively bi- ased in favor of recommendations men- tioning business deals and material things, even though such recommendations do not indeed prove any more useful objectively. 1 Introduction Finding good recommendations is an integral part of a modern information-seeking life – from pur- chasing products based on reviews to finding an- swers to baffling questions. Following the tradi- tion of sentiment analysis, many have proposed methods to automatically assess the quality of rec- ommendations or comments based on subjective ratings of their usefulness (Liu et al., 2007; Siers- dorfer et al., 2010; Becker et al., 2012; Momeni et al., 2013) or of persuasiveness (Wei et al., 2016). However, information thought to be useful does not always prove so, and subjective ratings may be driven by biases. Reviews which convince you to watch a movie or buy a product do not guarantee that you will enjoy the product, and the most con- vincing or highest rated answers to questions on sites like Stack Overflow or Yahoo Answers are not always the most accurate. We explore recommendations in a unique dataset, an online forecasting competition, which offers a rare glimpse into both subjective and ob- jective quality. In this competition, the users (forecasters) had a measurable goal — to fore- cast the outcomes of geopolitical events — and a need to effectively gather information in order to reach this goal. They viewed recommendations (or “tips”) from other forecasters, rated them, and po- tentially updated their own forecast. This data not only allows us to access what information the fore- casters thought useful based on their ratings, but also what was objectively useful based on (a) the rate at which forecasters change their prediction after viewing tips and, (b) the average improve- ment (or decrease) in their prediction accuracy af- ter this change. We seek to tease out subjective biases by dis- tinguishing the linguistic characteristics of rec- ommendations …

Content truncated to 500 words