Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 4873–4885 Barcelona, Spain (Online), December 8-13, 2020 4873 Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Models Meiqi Guo1, Rebecca Hwa1, Yu-Ru Lin2, Wen-Ting Chung3 1Department of Computer Science, University of Pittsburgh 2Department of Informatics and Networked Systems, University of Pittsburgh 3Department of Psychology in Education, University of Pittsburgh Pittsburgh, PA 15260, USA meiqi.guo@pitt.edu, hwa@cs.pitt.edu, {yurulin, wtchung}@pitt.edu Abstract We investigate the impact of political ideology biases in training data. Through a set of compar- ison studies, we examine the propagation of biases in several widely-used NLP models and its effect on the overall retrieval accuracy. Our work highlights the susceptibility of large, complex models to propagating the biases from human-selected input, which may lead to a deterioration of retrieval accuracy, and the importance of controlling for these biases. Finally, as a way to mitigate the bias, we propose to learn a text representation that is invariant to political ideology while still judging topic relevance. 1 Introduction Due to the extensive reaches of its network and the breadth of information enmeshed in it, social media has become an invaluable data source for empirical studies in the social sciences. Yet, identifying all and only relevant information out of a vast data stream remains an untamed challenge. While topic detection methods may help researchers extract some relevant text about a topic of interest (e.g., immigration policies), they may miss other equally relevant text while including some irrelevant ones. Crucially, because most topic detection methods are trained, they may unintentionally contain or propagate certain biases (e.g., extracting more instances written by women where gender balance is expected), resulting in a skewed data collection that may lead social scientists to draw incorrect conclusions. This paper explores the interactions between social biases and automatic topic detection models, and their impact on the resulting data collection. Our goal is to help social scientists gain insights about biases in text analytic so as to mitigate such biases in their data collections. More specifically, we examine the role of political ideology biases (liberal-leaning, denoted as Blue, and conservative-leaning, denoted as Red) in the process of collecting data about certain social topics (immigration and gun control) from Twitter. We observe that biases may be introduced at three major junctures of the data collection pipeline. First, it may be introduced in the data source itself (e.g., certain forums may have strong political leanings), but social scientists typically choose their data intentionally and are aware of pre-existing biases therein (Malik et al., 2015; Kosinski et al., 2015; Cihon and Yasseri, 2016). Second, biases may be introduced in the way in which “topic relevance” is defined. For example, domain experts may be consulted to identify a set of keywords or sample instances that are indicative or representative of the topic of interest. Thus, any unconscious bias on the part of the domain experts would be encoded into these keywords and examples (King et al., 2017), …

Content truncated to 500 words