Agree or Disagree: Predicting Judgments on Nuanced Assertions

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 214–224 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Agree or Disagree: Predicting Judgments on Nuanced Assertions Michael Wojatzki Torsten Zesch Language Technology Lab University of Duisburg-Essen, Germany michael.wojatzki@uni-due.de torsten.zeschi@uni-due.de Saif M. Mohammad Svetlana Kiritchenko National Research Council Canada Ottawa, Canada saif.mohammad@nrc-cnrc.gc.ca svetlana.kiritchenko@nrc-cnrc.gc.ca Abstract Being able to predict whether people agree or disagree with an assertion (i.e. an explicit, self- contained statement) has several applications ranging from predicting how many people will like or dislike a social media post to classi- fying posts based on whether they are in ac- cordance with a particular point of view. We formalize this as two NLP tasks: predicting judgments of (i) individuals and (ii) groups based on the text of the assertion and previ- ous judgments. We evaluate a wide range of approaches on a crowdsourced data set con- taining over 100,000 judgments on over 2,000 assertions. We find that predicting individual judgments is a hard task with our best results only slightly exceeding a majority baseline. Judgments of groups, however, can be more reliably predicted using a Siamese neural net- work, which outperforms all other approaches by a wide margin. 1 Introduction One of the most basic reactions when reading a sentence is to agree or disagree with it.1 Mech- anisms that allow us to express agreement (e.g. thumb-up, like, up-vote, ♥) or disagreement (e.g. thumb-down, dislike, down-vote) towards posts of other users can be found in almost all social networking sites. The judgments associated with posts that discuss controversial political or social issues, such as legalization of drug, immigration policy, or gun rights, are a rich source of infor- mation for those interested in the opinions of in- dividuals or groups. For instance, public opinion regarding an issue is often illustrated by the num- ber of retweets, likes, or upvotes that a politician or influential person receives. 1You are probably thinking about whether you agree with that statement right now. Hence, especially for controversial issues, be- ing able to predict how people judge posts has sev- eral applications: people at large could automati- cally anticipate if politicians, companies or other decision makers would agree or disagree with a new perspective on a problem or how they would evaluate a new possible solution. The method can also be used by journalists to more accurately ana- lyze the homogeneity of opinions or to detect filter bubbles in social media. Decision makers them- selves would be able to evaluate in advance how citizens, customers, or employees react to a press announcement, a new regulation, or tweet. Social media users could be enabled to search, sort or fil- ter posts based on whether they are in accordance with or contrary to their personal world view. Such predictions could also be used to augment chat ap- plications by indicating to a user if her recipients will agree or disagree with a message to be sent, enabling to choose a more or …

Content truncated to 500 words