Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview

Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview Deven Shah Stony Brook University dsshah@cs.stonybrook.edu H. Andrew Schwartz Stony Brook University has@cs.stonybrook.edu Dirk Hovy Bocconi University dirk.hovy@unibocconi.it Abstract An increasing number of natural language processing papers address the effect of bias on predictions, introducing mitigation tech- niques at different parts of the standard NLP pipeline (data and models). However, these works have been conducted individually, with- out a unifying framework to organize efforts within the field. This situation leads to repet- itive approaches, and focuses overly on bias symptoms/effects, rather than on their origins, which could limit the development of effective countermeasures. In this paper, we propose a unifying predictive bias framework for NLP. We summarize the NLP literature and suggest general mathematical definitions of predictive bias. We differentiate two consequences of bias: outcome disparities and error dispari- ties, as well as four potential origins of bi- ases: label bias, selection bias, model over- amplification, and semantic bias. Our frame- work serves as an overview of predictive bias in NLP, integrating existing work into a single structure, and providing a conceptual baseline for improved frameworks. 1 Introduction Predictive models in NLP are sensitive to a variety of (often unintended) biases throughout the devel- opment process. As a result, fitted models do not generalize well, incurring performance and relia- bility losses on unseen data. They also have so- cially undesirable effects by systematically under- serving or mispredicting certain user groups. The general phenomenon of biased predictive models in NLP is not recent. The community has long worked on the domain adaptation prob- lem (Jiang and Zhai, 2007; Daume III, 2007): models fit on newswire data do not perform well on social media and other text types. This prob- lem arises from the tendency of statistical mod- els to pick up on non-generalizable signals during the training process. In the case of domains, these non-generalizations are words, phrases, or senses that occur in one text type, but not another. However, this kind of variation is not just re- stricted to text domains: it is a fundamental prop- erty of human-generated language: we talk differ- ently than our parents or people from a different part of our country, etc. (Pennebaker and Stone, 2003; Eisenstein et al., 2010; Kern et al., 2016). In other words, language reflects the diverse de- mographics, backgrounds, and personalities of the people who use it. While these differences are of- ten subtle, they are distinct and cumulative (Trudg- ill, 2000; Kern et al., 2016; Pennebaker, 2011). Similar to text domains, this variation can lead models to pick up on patterns that do not gener- alize to other author-demographics, or to rely on undesirable word-demographic relationships. Bias may be an inherent property of any NLP system (and broadly any statistical model), but this is not per se negative. In essence, biases are priors that inform our decisions (a dialogue sys- tem designed for elders might work differently than one for teenagers). Still, undetected and unaddressed, biases can …

Content truncated to 500 words