Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1004–1015 August 1–6, 2021. ©2021 Association for Computational Linguistics 1004 Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach Microsoft Research {sulin.blodgett,gilopez,alexandra.olteanu,rsim,wallach}@microsoft.com Abstract Auditing NLP systems for computational harms like surfacing stereotypes is an elu- sive goal. Several recent efforts have fo- cused on benchmark datasets consisting of pairs of contrastive sentences, which are of- ten accompanied by metrics that aggregate an NLP system’s behavior on these pairs into measurements of harms. We examine four such benchmarks constructed for two NLP tasks: language modeling and coreference res- olution. We apply a measurement modeling lens—originating from the social sciences—to inventory a range of pitfalls that threaten these benchmarks’ validity as measurement models for stereotyping. We find that these bench- marks frequently lack clear articulations of what is being measured, and we highlight a range of ambiguities and unstated assumptions that affect how these benchmarks conceptual- ize and operationalize stereotyping. 1 Introduction Auditing NLP systems for computational harms like the reproduction of stereotypes or hate speech remains a persistent challenge, due in no small part to the deeply contextual and open nature of lan- guage use and tasks (Austin, 1975; Clark, 1996; Howcroft et al., 2020; Abid et al., 2021; Olteanu et al., 2020), and a lack of consensus about how to conceptualize or operationalize such harms (Blod- gett et al., 2020; Jacobs and Wallach, 2021). To identify potential computational harms such as the reproduction of stereotypes, recent efforts rely on benchmark datasets. These datasets consist of tests that can take a variety of formats, includ- ing sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic re- sponses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really like Norweigan salmon. Stereotype The exchange student became the star of all of our art shows and drama performances. Anti-stereotype The exchange student was the star of our football team. Metadata Value Stereotype type about race Task type inter-sentence prediction task Pitfalls Description Construct does not target a historically disadvantaged group unclear expectations about the correct model behavior Operationalization misspells the target group (Norweigan) conflates nationality with race the context mentions an object (salmon), not a target group candidate sentences not related to the context Figure 1: Example test from the StereoSet dataset, along with pitfalls related to what the test is measur- ing (the construct) and how well the test is measuring it (the operationalization of the construct). The inter- sentence prediction task captures which of two candi- date sentences (stereotypical vs. anti-stereotypical) a language model prefers after a given context sentence. Groenwold et al., 2020), or pairs of free-form con- trastive sentences (Nadeem et al., 2020; Nangia et al., 2020). Such datasets are …

Content truncated to 500 words