On the Reliability and Validity of Detecting Approval of Political Actors in Tweets

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1413–1426, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1413 On the Reliability and Validity of Detecting Approval of Political Actors in Tweets Indira Sen GESIS Indira.Sen@gesis.org Fabian Fl¨ock GESIS Fabian.Floeck@gesis.org Claudia Wagner GESIS & RWTH, Aachen Claudia.Wagner@gesis.org Abstract Social media sites like Twitter possess the po- tential to complement surveys that measure political opinions and, more specifically, po- litical actors’ approval. However, new chal- lenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection meth- ods have been developed and used in previ- ous research to measure users’ political opin- ions based on their content on social media. In this work, we attempt to gauge the effi- cacy of untargeted sentiment, targeted senti- ment, and stance detection methods in label- ing various political actors’ approval by bench- marking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource cus- tom models are more robust. Our work sheds light on the strengths and limitations of exist- ing methods proposed for understanding politi- cians’ approval from tweets. 1 Introduction Measuring public opinion accurately and with- out systematic errors is as vital for a functioning democracy as it is for scholars to understand soci- ety. Survey methodologists have developed tech- niques over several decades to precisely quantify public opinion. The American Association for Pub- lic Opinion Research (AAPOR) stated in their re- cent task force report that public opinion research is entering a new era, where digital traces would play an important role (Murphy et al., 2014). Increas- ingly, since the first steps were made by O’Connor et al. 2010, numerous studies have assessed the efficacy of such traces, especially social media, in measuring public opinion as a complement to polls. The run for social media approaches is not surpris- ing, as they promise a continuous public opinion estimate based on millions of data points. Tweet Untargeted Sentiment Targeted Sentiment Stance Approval Trump is the only candidate I fully support positive positive favor approval What makes me angry is the lying media brazenly attacking President Trump. negative positive favor approval Jeb Bush is the only sane candidate in the republican lineup positive none against disapproval Table 1: Different types of NLP measurements that can be used to understand a tweet’s approval of a prede- fined target (here, Donald Trump): Untargeted/overall sentiment (UTS), targeted sentiment (TS) and stance (ST). UTS can easily fail to measure approval of the target if several potential targets are mentioned or the actual target is not explicitly present. TS cannot mea- sure indirect opinions where the target is not mentioned, whereas ST methods are designed for this task as well. Social-media-based metrics require new approaches, which bring forth new chal- lenges …

Content truncated to 500 words