Intrinsic Bias Metrics Do Not Correlate with Application Bias

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1926–1940 August 1–6, 2021. ©2021 Association for Computational Linguistics 1926 Intrinsic Bias Metrics Do Not Correlate with Application Bias Seraphina Goldfarb-Tarrant∗† Rebecca Marchant∗† Ricardo Mu˜noz S´anchez∗† Mugdha Pandya∗† Adam Lopez‡† †University of Edinburgh, ‡Rasa Technologies GmbH s.tarrant@ed.ac.uk {rebecca.marchant31, ricardoms.math, pandya.mugdha4}@gmail.com a.lopez@rasa.com Abstract Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP commu- nity relies on a variety of metrics that quan- tify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic met- rics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and lan- guages. We urge researchers working on de- biasing to focus on extrinsic measures of bias, and to make using these measures more feasi- ble via creation of new challenge sets and an- notated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.1 1 Introduction Awareness of bias in Natural Language Processing (NLP) systems has rapidly increased as more and more systems are discovered to perpetuate societal unfairness at massive scales. This awareness has prompted a surge of research into measuring and mitigating bias, but this research suffers from lack of consistent metrics that discover and measure bias. Instead, work on bias is “rife with unstated assumptions” (Blodgett et al., 2020) and relies on metrics that are easy to measure rather than metrics that meaningfully detect bias in applications. ∗ Equal contribution. Correspondence to s.tarrant@ ed.ac.uk 1https://tinyurl.com/serif-embed (a) Intrinsic metrics summarize biases in the geometry of embeddings. For example, in this embedding space, male words are closer to words about career and about math & science, whereas female words are closer to words about family. (b) Extrinsic bias metrics summarize disparities in appli- cation performance across populations, such as rates of false negatives between different gender groups. For ex- ample, a coreference system may make more errors in an anti-stereotypical career coreferent (red arc) than in a pro-stereotypical one (green arc). Figure 1: The relationship between intrinsic bias met- rics (a) and extrinsic bias metrics (b) has been assumed, but not confirmed. A recent comprehensive survey of bias in NLP (Blodgett et al., 2020) found that one third of all re- search papers focused on bias in word embeddings. This makes embeddings the most common topic in studies of bias — over twice as common as any other topic related to bias in NLP. As is visualised in Figure 1a, bias …

Content truncated to 500 words