Classification without (Proper) Representation: Political Heterogeneity in Social Media and Its Implications for Classification and Behavioral Analysis

Findings of the Association for Computational Linguistics: ACL 2022, pages 504 - 522 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Classification without (Proper) Representation: Political Heterogeneity in Social Media and Its Implications for Classification and Behavioral Analysis Kenan Alkiek University of Michigan kalkiek@umich.edu Bohan Zhang University of Michigan zbohan@umich.edu David Jurgens University of Michigan jurgens@umich.edu Abstract Reddit is home to a broad spectrum of political activity, and users signal their political affilia- tions in multiple ways—from self-declarations to community participation. Frequently, com- putational studies have treated political users as a single bloc, both in developing models to infer political leaning and in studying polit- ical behavior. Here, we test this assumption of political users and show that commonly- used political-inference models do not gener- alize, indicating heterogeneous types of politi- cal users. The models remain imprecise at best for most users, regardless of which sources of data or methods are used. Across a 14-year longitudinal analysis, we demonstrate that the choice in definition of a political user has sig- nificant implications for behavioral analysis. Controlling for multiple factors, political users are more toxic on the platform and inter-party interactions are even more toxic—but not all political users behave this way. Last, we iden- tify a subset of political users who repeatedly flip affiliations, showing that these users are the most controversial of all, acting as provoca- teurs by more frequently bringing up politics, and are more likely to be banned, suspended, or deleted. 1 Introduction Individuals readily engage in political behavior online, sharing content and forming communities with like-minded individuals. Scholars study these active political communities to understand parti- sanship (Leong et al., 2020), polarization (Morales et al., 2021; Hofmann et al., 2021), and voting be- haviors (Gayo-Avello, 2012). Many computational studies of political behav- ior in social media have the underlying assumption that political leanings can be reliably identified. Prior work has shown that partisan leaning can be inferred from a diverse set of behavioral charac- teristics such as text (Volkova et al., 2014), social networks (Lindamood et al., 2009; Barberá, 2015), and even community participation (An et al., 2019). Yet, inferring political leaning is known to be a challenging problem (Cohen and Ruths, 2013), in particular for centrist or apolitical users who infre- quently express political beliefs. Further, inference models typically use a single source of information on a user’s political affiliation without examining whether this source generalizes to all types of users. This methodology fails to account for the disparate types of political users and introduces sampling bias downstream. Here, we re-examine inferring political behavior for these diverse groups in a uni- fied setting to understand the consequences our data have on results. This paper tests what effect current assumptions of social media users’ political affiliations have on our ability to model political users and their behaviors. The first part of the paper tests how different definitions of political users generalize to other users’ behaviors and to inferring political leaning. Using 574K political users on Reddit, …

Content truncated to 500 words