Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3798–3808 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3798 Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion Flora Sakketou†, Allison Lahnala†, Liane Vogel‡, Lucie Flek† †Conversational AI and Social Analytics (CAISA) Lab Department of Mathematics and Computer Science, University of Marburg ‡Data Management Lab, Department of Computer Science, Technical University of Darmstadt {flora.sakketou, allison.lahnala, lucie.flek}@uni-marburg.de, liane.vogel@cs.tu-darmstadt.de Abstract There is an increasing need for the ability to model fine-grained opinion shifts of social media users, as concerns about the potential polarizing social effects increase. However, the lack of publicly available datasets that are suitable for the task presents a major challenge. In this paper, we introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads, thus making subtle opinion fluctuations detectable both in long term and in short term. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts. We provide a strategy for recruiting suitable non-experts. Our analysis of the inter-annotator agreements shows that the resulting annotations obtained from the majority vote of the non-experts are of comparable quality to the annotations of the experts. We provide analyses of the stance evolution in short term and long term levels, a comparison of language usage between users with vacillating and resolute attitudes, and fine-grained stance detection baselines. Keywords: opinion dynamics, stance detection dataset, sociopolitical language 1. Introduction Social media networks play a major role in platforming the expression of opinions on innumerable topics. With concerns growing about how they may facilitate “echo- chambers” that could could amplify polarization and radicalization (Vicario et al., 2016; Strandberg et al., 2018; M¨as et al., 2013; M¨as and Flache, 2013), tem- poral data about user opinions is necessary in order to study the dynamics of these spaces and their potential to radicalize individuals (Matthes, 2014). While there is abundant research in sociopolitical sci- ences regarding opinion formation and expression, there is a significant lack of Natural Language Pro- cessing (NLP) data resources that enable the study of temporal opinion dynamics, thus also, a lack of effi- cient, data-driven models that investigate the relation- ships between social media behaviors and opinion for- mation and fluctuations (Dowling et al., 2020a; Ace- moglu and Ozdaglar, 2011). Meanwhile, the increasing number of people using social media platforms makes the need for models that analyze and forecast public opinion all the more vital. A relevant body of work concerns automatic stance detection (Mohammad et al., 2016b; Walker et al., 2012b; Somasundaran and Wiebe, 2010; Thomas et al., 2006), the task of determining the polarity of a stance toward a given subject, for which many datasets have been developed. These datasets however do not satisfy the needs for studying opinion dynamics from text. One …

Content truncated to 500 words