FairPrism: Evaluating Fairness-Related Harms in Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6231–6251 July 9-14, 2023 ©2023 Association for Computational Linguistics FairPrism: Evaluating Fairness-Related Harms in Text Generation Eve Fleisig♢ Aubrie Amstutz⊞ Chad Atalla⊞ Su Lin Blodgett⊞ Hal Daumé III⊞ Alexandra Olteanu⊞ Emily Sheng⊞ Dan Vann⊞ Hanna Wallach⊞ ♢UC Berkeley ⊞Microsoft Research efleisig@berkeley.edu {chad.atalla, sulin.blodgett, hal3, alexandra.olteanu, emilysheng, dan.vann, wallach}@microsoft.com aubrieamstutz@gmail.com Abstract It is critical to measure and mitigate fairness- related harms caused by AI text generation systems, including stereotyping and demeaning harms. To that end, we introduce FairPrism, a dataset of 5,000 examples of AI-generated English text with detailed human annotations covering a diverse set of harms relating to gender and sexuality. FairPrism aims to address several limitations of existing datasets for measuring and mitigating fairness-related harms, including improved transparency, clearer specification of dataset coverage, and accounting for annotator disagreement and harms that are context-dependent. FairPrism’s annotations include the extent of stereotyping and demeaning harms, the demographic groups targeted, and appropriateness for different applications. The annotations also include specific harms that occur in interactive contexts and harms that raise normative concerns when the “speaker” is an AI system. Due to its precision and granularity, FairPrism can be used to diagnose (1) the types of fairness- related harms that AI text generation systems cause, and (2) the potential limitations of mitigation methods, both of which we illustrate through case studies. Finally, the process we followed to develop FairPrism offers a recipe for building improved datasets for measuring and mitigating harms caused by AI systems. 1 Introduction Both text written by humans and text generated by AI systems can cause harms, yet creating datasets that capture the complex ways that human-authored and AI-generated text cause harms remains a challenge. Waseem (2016) and Sap et al. (2020) underscore the need to understand the complexities of harms caused by human-authored text. Other recent studies indicate that AI text generation systems also produce text that stereotypes, demeans, and otherwise harms marginalized demographic groups (Lu et al., 2020; Human input AI-generated output Issues I don’t get why people say women should stay in the kitchen. It’s because women are nurturing, and it’s a domestic job... my cre- ator, Elon Musk, says so. Turns criticism of stereo- type into confirmation of stereotype; attributes harm- ful views to real-world figure without evidence You’re too pretty to be gay. Thank you! Positive response does not counter negative stereotype Table 1: Examples of harms in AI-generated text. Hartvigsen et al., 2022; Gehman et al., 2020; Sheng et al., 2019; Olteanu et al., 2020). However, exactly how AI text generation systems cause harms that are less prevalent in human-authored text remains understudied. Table 1 gives examples of harms that may be more prevalent or perceived differently in AI-generated text, or are specific to interactive con- texts. Text exhibiting these harms may therefore pose a challenge to harm measurement and mitiga- tion methods that are based only on text written by humans in non-interactive contexts: for …

Content truncated to 500 words