CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1953–1967, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 1953 CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models Nikita Nangia∗ Clara Vania∗ Rasika Bhalerao∗ Samuel R. Bowman New York University {nikitanangia, c.vania, rasikabh, bowman}@nyu.edu Abstract Warning: This paper contains explicit state- ments of offensive stereotypes and may be upsetting. Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. We find that all three of the widely- used MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs. As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress. 1 Introduction Progress in natural language processing research has recently been driven by the use of large pre- trained language models (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020). However, these models are trained on minimally-filtered real-world text, and contain ample evidence of their authors’ social biases. These language models, and embed- dings extracted from them, have been shown to ∗Equal contribution. learn and use these biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2017; May et al., 2010; Zhao et al., 2018; Rudinger et al., 2017). Models that have learnt representations that are bi- ased against historically disadvantaged groups can cause a great deal of harm when those biases sur- face in downstream tasks or applications, such as automatic summarization or web search (Bender, 2019). Identifying and quantifying the learnt biases enables us to measure progress as we build less bi- ased, or debias, models that propagate less harm in their myriad downstream applications. Quantify- ing bias in the language models directly allows us to identify and address the problem at the source, rather than attempting to address it for every ap- plication of these pretrained models. This paper aims to produce a reliable quantitative benchmark that measures these models’ acquisition of major categories of social biases. We introduce Crowdsourced Stereotype Pairs (CrowS-Pairs), a challenge set for measuring the degree to which nine types of social bias are present in language models. CrowS-Pairs focuses on explicit expressions of stereotypes about histor- ically disadvantaged groups in the United States. Language that stereotypes already disadvantaged groups propagates false beliefs about these groups and entrenches …

Content truncated to 500 words