NLPositionality: Characterizing Design Biases of Datasets and Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9080–9102 July 9-14, 2023 ©2023 Association for Computational Linguistics NLPositionality: Characterizing Design Biases of Datasets and Models Sebastin Santy†∗ Jenny T. Liang‡∗ Ronan Le Bras⋄ Katharina Reinecke† Maarten Sap‡⋄ †University of Washington ‡Carnegie Mellon University ⋄Allen Institute for AI {ssanty,reinecke}@cs.washington.edu, {jtliang,maartensap}@cs.cmu.edu, ronanlb@allenai.org Abstract Design biases in NLP systems, such as perfor- mance differences for different populations, of- ten stem from their creator’s positionality, i.e., views and lived experiences shaped by identity and background. Despite the prevalence and risks of design biases, they are hard to quan- tify because researcher, system, and dataset positionality is often unobserved. We intro- duce NLPositionality, a framework for char- acterizing design biases and quantifying the positionality of NLP datasets and models. Our framework continuously collects annotations from a diverse pool of volunteer participants on LabintheWild, and statistically quantifies alignment with dataset labels and model predic- tions. We apply NLPositionality to existing datasets and models for two tasks—social ac- ceptability and hate speech detection. To date, we have collected 16, 299 annotations in over a year for 600 instances from 1, 096 annota- tors across 87 countries. We find that datasets and models align predominantly with Western, White, college-educated, and younger popula- tions. Additionally, certain groups, such as non- binary people and non-native English speakers, are further marginalized by datasets and mod- els as they rank least in alignment across all tasks. Finally, we draw from prior literature to discuss how researchers can examine their own positionality and that of their datasets and models, opening the door for more inclusive NLP systems. 1 Introduction “Treating different things the same can generate as much inequality as treating the same things differently.” – Kimberlé Crenshaw When creating NLP datasets and models, re- searchers’ design choices are partly influenced * Equal contribution; work done while at the Allen Insti- tute for AI 1 2 3 4 5 6 7 ✓ x ✓ ✓ x x ✓ Hate Speech Database 1 2 3 4 5 6 7 0.95 0.25 0.30 0.82 0.32 0.67 0.27 Toxic101 Dataset • Quality annotations only from master turkers, better if educated. • Maybe only English phrases? • Are micro-aggressions toxic? And misogynistic comments? • Should I include gender stereotypes? Researcher from USA ! Woman Activist & Carl Jones, Tech Lead, New York Times Aditya Sharma, Tech Lead, Times of India Developers use PerspectiveAPI to remove toxic comments from their news platforms • “These people are full of shit” (0.67) • “Can you stop being a jerk?” (0.82) • “Presstitutes everywhere on the news!” (0.33) • “These jihadis need to be ousted” (0.41) Design Choices Researcher Positionality …. …. …. Dataset Positionality Performance mismatch for different populations Model Positionality (Latent) (Latent) Figure 1: Example Scenario. Carl from the U.S. and Aditya from India both want to use Perspective API, but it works better for Carl than it does for Aditya. This is because toxicity researchers’ positionalities lead them to …

Content truncated to 500 words