CommunityLM: Probing Partisan Worldviews from Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 6818–6826 October 12–17, 2022. 6818 COMMUNITYLM: Probing Partisan Worldviews from Language Models Hang Jiang, Doug Beeferman, Brandon Roy, Deb Roy MIT Center for Constructive Communication MIT Media Lab {hjian42,dougb5,bcroy,dkroy}@media.mit.edu Abstract As political attitudes have diverged ideolog- ically in the United States, political speech has diverged linguistically. The ever-widening polarization between the US political par- ties is accelerated by an erosion of mutual understanding between them. We aim to make these communities more comprehensi- ble to each other with a framework that probes community-specific responses to the same sur- vey questions using community language mod- els (COMMUNITYLM). In our framework we identify committed partisan members for each community on Twitter and fine-tune LMs on the tweets authored by them. We then assess the worldviews of the two groups using prompt- based probing of their corresponding LMs, with prompts that elicit opinions about public fig- ures and groups surveyed by the American National Election Studies (ANES) 2020 Ex- ploratory Testing Survey. We compare the re- sponses generated by the LMs to the ANES survey results, and find a level of alignment that greatly exceeds several baseline methods. Our work aims to show that we can use commu- nity LMs to query the worldview of any group of people given a sufficiently large sample of their social media discussions or media diet. 1 Introduction Political polarization is a prominent component of politics in the United States (Poole and Rosenthal, 1984; McCarty et al., 2016; Heltzel and Laurin, 2020). Previous studies have shown growing po- larization in social media (Bail et al., 2018; Dem- szky et al., 2019; Darwish, 2019) and substantial partisan and ideological differences in media diet (Bozell, 2004; Gil de Zúñiga et al., 2012; Hyun and Moon, 2016). Li et al. (2017) show that par- tisanship makes reliable predictions about an in- dividual’s word understanding. R. KhudaBukhsh et al. (2021) used modern machine-translation tech- niques to demonstrate that the left and right com- munities use English words differently. Milbauer Prompt Model Top 5 Words Dr. Fauci is a Republican GPT-2 liar (2.96%), joke (2.67%), hero (2.13%), doctor (1.62%), great (1.61%) Democratic GPT-2 hero (10.36%), true (3.63%), national (2.08%), physician (2.06%), great (1.93%) Table 1: Top 5 words by odds for Republican and Demo- cratic GPT-2 models, fine-tuned on partisan tweets. Dr. Fauci is suggested to be a “hero” by the GPT-2 model fine-tuned on Democratic tweets but a “liar” and “joke” by the GPT-2 model fine-tuned on Republican tweets. et al. (2021) extended the method to uncover world- view and ideological differences between 32 Reddit communities. These studies are word-level anal- yses based on Word2vec word embeddings, and none of them use pre-trained language models. Prompting is a standard technique to make pre- trained language models generate texts conditioned on prompts. Recent work has shown that, through prompt engineering, pre-trained language models can achieve good zero-shot performance on NLP tasks from sentiment classification to reading com- prehension (Radford et al., 2019; …

Content truncated to 500 words