Leveraging Order-Free Tag Relations for Context-Aware Recommendation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 312–324 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 312 (Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys Kenneth Joseph University at Buffalo Buffalo, NY, USA kjoseph@buffalo.edu Sarah Shugars New York University New York, NY, USA sarah.shugars@nyu.edu Ryan Gallagher and Jon Green and Alexi Quintana Mathé Northeastern University Boston, MA, USA gallagher.r,jo.green,quintanamatha.a@northeastern.edu Zijian An University at Buffalo Buffalo, NY, USA zijianan@buffalo.edu David Lazer Northeastern University Boston, MA, USA d.lazer@northeastern.edu Abstract Stance detection, which aims to determine whether an individual is for or against a tar- get concept, promises to uncover public opin- ion from large streams of social media data. Yet even human annotation of social media content does not always capture “stance” as measured by public opinion polls. We demon- strate this by directly comparing an individual’s self-reported stance to the stance inferred from their social media data. Leveraging a longitu- dinal public opinion survey with respondent Twitter handles, we conducted this compari- son for 1,129 individuals across four salient targets. We find that recall is high for both “Pro” and “Anti” stance classifications but pre- cision is variable in a number of cases. We identify three factors leading to the disconnect between text and author stance: temporal incon- sistencies, differences in constructs, and mea- surement errors from both survey respondents and annotators. By presenting a framework for assessing the limitations of stance detection models, this work provides important insight into what stance detection truly measures. 1 Introduction While surveys have a long history of quantifying public opinion (Berinsky, 2017), deploying them longitudinally requires considerable resources and may produce measurements which lag behind rapidly evolving events. Particularly during on- going or developing situations, such as the COVID- 19 pandemic and U.S. presidential elections, accu- rately capturing both current and historical opinion may be crucial to crafting appropriate policy in- terventions or anticipating political outcomes. In contrast to surveys, stance detection algorithms, which identify whether a given text is in favor or against a target concept, can be easily deployed at scale on social media data (Aldayel and Magdy, 2021), positioning them as a potential supplement or replacement for traditional public opinion sur- veys (Küçük and Can, 2020). However, stance detection methods have rarely been evaluated in comparison to the public opinion data they promise to replace. Instead, stance de- tection methods have almost exclusively been eval- uated through annotated social media data, often produced by crowdworkers. When disagreements in stance occur across annotators or tasks, existing stance detection work often chooses to either ig- nore the disagreements (Mohammad et al., 2016; Sobhani et al., 2017; Samih and Darwish, 2021), reconcile them with expert annotators (van den Berg et al., 2019), and/or account for them by mod- eling the annotation (Joseph et al., 2017; van den Berg et al., 2019). This leaves two critical, largely open questions for stance detection. First, when do stance annota- tors agree (RQ1)? Annotating stance is a subjective …

Content truncated to 500 words