What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1762–1771 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1762 What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology Qinlan Shen Carnegie Mellon University qinlans@cs.cmu.edu Carolyn P. Ros´e Carnegie Mellon University cprose@cs.cmu.edu Abstract In this paper, we challenge the assumption that political ideology is inherently built into text by presenting an investigation into the im- pact of experiential factors on annotator per- ceptions of political ideology. We construct an annotated corpus of U.S. political discus- sion, where in addition to ideology labels for texts, annotators provide information about their political affiliation, exposure to politi- cal news, and familiarity with the source do- main of discussion, Reddit. We investigate the variability in ideology judgments across annotators, finding evidence that these experi- ential factors may influence the consistency of how political ideologies are perceived. Finally, we present evidence that understanding how humans perceive and interpret ideology from texts remains a challenging task for state-of- the-art language models, pointing towards po- tential issues when modeling user experiences that may require more contextual knowledge. 1 Introduction Social media companies, like Twitter, Facebook, and Reddit, play an important role in political dis- course by providing a space for users to interact with different viewpoints. Understanding political discussion on these platforms often requires one to identify the ideologies behind texts, as understand- ing the viewpoints reflected in a text can provide insight into the partisanship of beliefs (Monroe et al., 2008) or the persuasive strategies used by different ideological groups (Tsur et al., 2015). Prior research on political discussion often relies on a ”ground-truth” to aid in obtaining ideology la- bels for social media data. For example, due to the scale of political content on social media, a com- mon paradigm is to obtain some ground-truth labels that are propagated to a larger set of texts using semi-supervised learning (Lin and Cohen, 2010; Zhou et al., 2011). The relationship between a so- cial media artifact and various forms of established political knowledge can also be used to ground or validate ideology labels. Some examples of this include using author interactions with politicians with known party affiliations (Djemili et al., 2014; Barber´a, 2015), ideological communities (Chan- drasekharan et al., 2017; Shen and Ros´e, 2019), and central users (Pennacchiotti and Popescu, 2011; Preotiuc-Pietro et al., 2017) as a starting heuristic, or evaluating a labeling approach by comparing geolocation tags attached to posts with historical voting patterns (Demszky et al., 2019). A limitation of these approaches, however, is that behavior on social media does not evenly or uniformly reflect the held political beliefs of par- ticipants. While there is evidence that people tend to engage with others who share similar beliefs (Halberstam and Knight, 2016), people also com- monly interact with or even seek out communi- ties and users they do not agree with (Kelly et al., 2005; Tan et al., 2016). Additionally, the practice of displaying …

Content truncated to 500 words