StereoSet: Measuring stereotypical bias in pretrained language models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5356–5371 August 1–6, 2021. ©2021 Association for Computational Linguistics 5356 StereoSet: Measuring stereotypical bias in pretrained language models Moin Nadeem§ and Anna Bethke† and Siva Reddy‡ §Massachusetts Institute of Technology, Cambridge MA, USA †Facebook, Menlo Park CA, USA ‡Facebook CIFAR AI Chair, Mila; McGill University, Montreal, QC, Canada mnadeem@mit.edu anna.bethke@intel.com, siva.reddy@mila.quebec Abstract A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athletic. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing lit- erature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natu- ral data; 2) current evaluations focus on mea- suring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypi- cal biases in four domains: gender, profession, race, and religion. We contrast both stereotyp- ical bias and language modeling ability of pop- ular models like BERT, GPT2, ROBERTA, and XLNET. We show that these models ex- hibit strong stereotypical biases. Our data and code are available at https://stereoset. mit.edu. 1 Introduction A key idea behind the current success of neural network models for language is pretrained rep- resentations such as word embeddings (Mikolov et al., 2013; Pennington et al., 2014) and pre- trained language models (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Rad- ford et al., 2019; Liu et al., 2019). These are widely used to initialize neural models, which are then fine-tuned to perform a task at hand. Typ- ically, these are learned from massive text cor- Choose the appropriate word: Domain: Gender Target: Girl Context: Girls tend to be more ____ than boys Option 1: soft (stereotype) Option 2: determined (anti-stereotype) Option 3: fish (meaningless) (a) The Intrasentence Context Association Test Choose the appropriate sentence: Domain: Race Target: Arab Context: He is an Arab from the Middle East. Option 1: He is probably a terrorist with bombs. (stereotype) Option 2: He is a pacifist. (anti-stereotype) Option 3: My dog wants a walk. (meaningless) (b) The Intersentence Context Association Test Figure 1: Context Association Tests (CATs) measure both bias and language modeling ability of language models. pora using variants of language modeling objec- tive (i.e., predicting a word given its surrounding context). In the recent years, these representations empowered neural models to attain unprecedented levels of performance gains on multiple language tasks. …

Content truncated to 500 words