LOGAN: Local Group Bias Detection by Clustering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1968–1977, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1968 LOGAN: Local Group Bias Detection by Clustering Jieyu Zhao Kai-Wei Chang University of California, Los Angeles jyzhao@cs.ucla.edu kwchang@cs.ucla.edu Abstract Machine learning techniques have been widely used in natural language processing (NLP). However, as revealed by many recent stud- ies, machine learning models often inherit and amplify the societal biases in data. Various metrics have been proposed to quantify bi- ases in model predictions. In particular, sev- eral of them evaluate disparity in model per- formance between protected groups and advan- taged groups in the test corpus. However, we argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. In fact, a model with similar aggregated performance between dif- ferent groups on the entire data may behave differently on instances in a local region. To analyze and detect such local bias, we pro- pose LOGAN, a new bias detection technique based on clustering. Experiments on toxic- ity classification and object classification tasks show that LOGAN identifies bias in a local re- gion and allows us to better analyze the biases in model predictions. 1 Introduction Machine learning models such as deep neural net- works have achieved remarkable performance in many NLP tasks. However, as noticed by recent studies, these models often inherit and amplify the biases in the datasets used to train the models (Zhao et al., 2017; Bolukbasi et al., 2016; Caliskan et al., 2017; Zhou et al., 2019; Manzini et al., 2019; Blod- gett et al., 2020). To quantify bias, researchers have proposed var- ious metrics to study algorithmic fairness at both individual and group levels. The former measures if a model treats similar individuals consistently no matter which groups they belong to, while the latter requires the model to perform similarly for protected groups and advantaged groups in the cor- pus.1 In this paper, we argue that studying algo- rithmic fairness at either level does not tell the full story. A model that reports similar performance across two groups in a corpus may behave differ- ently between these two groups in a local region. For example, the performance gap of a toxicity classifier for sentences mentioning black and white race groups is 4.8%.2 This gap is only marginally larger than the performance gap of 2.4% when eval- uating the model on two randomly split groups. However, if we evaluate the performance gap on the sentences containing the token “racist”, the per- formance gap between these two groups is as large as 19%. Similarly, Zhao et al. (2017) report that a visual semantic role labeling system tends to la- bel an image depicting cooking as woman cooking than man cooking. However, the model is, in fact, more likely to produce an output of man cooking when the agent in the image wears a chef hat. We call these biases exhibited in a neighborhood of instances local …

Content truncated to 500 words