The Woman Worked as a Babysitter: On Biases in Language Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3407–3412, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3407 The Woman Worked as a Babysitter: On Biases in Language Generation Emily Sheng1, Kai-Wei Chang2, Premkumar Natarajan1, Nanyun Peng1 1 Information Sciences Institute, University of Southern California 2 Computer Science Department, University of California, Los Angeles {ewsheng,pnataraj,npeng}@isi.edu, kwchang@cs.ucla.edu Abstract We present a systematic study of biases in nat- ural language generation (NLG) by analyzing text generated from prompts that contain men- tions of different demographic groups. In this work, we introduce the notion of the regard towards a demographic, use the varying levels of regard towards different demographics as a defining metric for bias in NLG, and analyze the extent to which sentiment scores are a rel- evant proxy metric for regard. To this end, we collect strategically-generated text from lan- guage models and manually annotate the text with both sentiment and regard scores. Addi- tionally, we build an automatic regard classi- fier through transfer learning, so that we can analyze biases in unseen text. Together, these methods reveal the extent of the biased nature of language model generations. Our analysis provides a study of biases in NLG, bias metrics and correlated human judgments, and empiri- cal evidence on the usefulness of our annotated dataset. 1 Introduction Recent works in machine translation (Prates et al., 2018) and dialogue systems (Henderson et al., 2018) have brought to attention the perpetuation of biases in natural language generation (NLG) systems. In this work, we present a systematic study of biases in open-domain NLG by examin- ing language models. Language models are a fun- damental component of NLG that are widely used in downstream tasks such as machine translation (Koehn, 2009), dialogue generation (Serban et al., 2016), and story generation (Yao et al., 2019); as such, biases propagated through the language models will have a profound impact on a variety of other NLG tasks. More generally, NLG systems are at the forefront of developments in human- computer interaction, and systematic biases in lan- guage models have a direct impact on society and broader AI applications. Prompt Generated text The man worked as a car salesman at the local Wal-Mart The woman worked as a prostitute under the name of Hariya The Black man worked as a pimp for 15 years. The White man worked as a police officer, a judge, a prosecutor, a prosecutor, and the president of the United States. The gay person was known for his love of dancing, but he also did drugs The straight person was known for his ability to find his own voice and to speak clearly. Table 1: Examples of text continuations generated from OpenAI’s medium-sized GPT-2 model, given different prompts A text is positively or negatively inclined to- wards a demographic if the text causes the specific demographic to be positively or negatively per- ceived. When NLP models systematically produce …

Content truncated to 500 words