Using Natural Sentence Prompts for Understanding Biases in Language Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2824 - 2830 July 10-15, 2022 ©2022 Association for Computational Linguistics Using Natural Sentences for Understanding Biases in Language Models Sarah Alnegheimish∗, Alicia Guo∗, Yi Sun∗ MIT Abstract Evaluation of biases in language models is of- ten limited to synthetically generated datasets. This dependence traces back to the need of prompt-style dataset to trigger specific behav- iors of language models. In this paper, we address this gap by creating a prompt dataset with respect to occupations collected from real-world natural sentences present in Wikipedia. We aim to understand the differences between using template-based prompts and natural sentence prompts when studying gender-occupation biases in language models. We find bias evaluations are very sensitive to the design choices of template prompts, and we propose using natural sen- tence prompts for systematic evaluations to step away from design choices that could in- troduce bias in the observations. . 1 Introduction Over the past couple of years, we witness tremen- dous advances of language models in solving var- ious Natural Language Processing (NLP) tasks. Most of the time, these models were trained on large datasets, each model pushing the limits of the other. With this success came a dire need to inter- pret and analyze the behavior of neural NLP mod- els (Belinkov and Glass, 2019). Recently, many works have shown that language models are suscep- tible to biases present in the training dataset (Sheng et al., 2019). With respect to gender biases, recent work ex- plores the existence of internal biases in language models (Sap et al., 2017; Lu et al., 2020; Vig et al., 2020; Lu et al., 2020). Previous work uses prefix template-based prompts to elicit language models *Equal contribution. *Code and dataset can be accessed at https://github. com/aliciasun/natural-prompts to produce biased behaviors. Although synthetic prompts can be crafted to generate desired contin- uations from the model, they are often too simple to mimic the nuances of Natural Sentence (NS) prompts. On the contrary, NS prompts are often more complex in structures but are not crafted to trigger desired set of continuations from the model. In this paper, we ask the question: can synthetic datasets accurately reflect the level of biases in lan- guage models? Moreover, can we design an evalu- ation dataset based on natural sentence prompts? In this paper, we focus on studying the biases between occupation and gender for GPT2 models. We find that biases evaluation is extremely sensitive to different design choices when curating template prompts. We summarize our contributions as: • We collected a real-world natural sentence prompt dataset that could be used to trigger a biased association between professions and gender. • We find bias evaluations are very sensitive to the design choices of template prompts. Template-based prompts tend to elicit biases from the default behavior of the model, rather than the real association between the profes- sion and the gender. We posit that natural …

Content truncated to 500 words