Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1477–1491 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1477 Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? Rochelle Choenni University of Amsterdam r.m.v.k.choenni@uva.nl Ekaterina Shutova University of Amsterdam e.shutova@uva.nl Robert van Rooij University of Amsterdam r.a.m.vanrooij@uva.nl Abstract Warning: this paper contains content that may be offensive or upsetting. In this paper, we investigate what types of stereotypical information are captured by pre- trained language models. We present the first dataset comprising stereotypical attributes of a range of social groups and propose a method to elicit stereotypes encoded by pretrained language models in an unsupervised fashion. Moreover, we link the emergent stereotypes to their manifestation as basic emotions as a means to study their emotional effects in a more generalized manner. To demonstrate how our methods can be used to analyze emotion and stereotype shifts due to linguistic experi- ence, we use fine-tuning on news sources as a case study. Our experiments expose how attitudes towards different social groups vary across models and how quickly emotions and stereotypes can shift at the fine-tuning stage. 1 Introduction Pretraining strategies for large-scale language mod- els (LMs) require unsupervised training on large amounts of human generated text data. While highly successful, these methods come at the cost of interpretability as it has become increasingly un- clear what relationships they capture. Yet, as their presence in society increases, so does the impor- tance of recognising the role they play in perpetu- ating social biases. In this regard, Bolukbasi et al. (2016) first discovered that contextualized word representations reflect gender biases captured in the training data. What followed was a suite of stud- ies that aimed to quantify and mitigate the effect of harmful social biases in word (Caliskan et al., 2017) and sentence encoders (May et al., 2019). Despite these studies, it has remained difficult to define what constitutes “bias”, with most work fo- cusing on “gender bias” (Manela et al., 2021; Sun et al., 2019) or “racial bias” (Davidson et al., 2019; Sap et al., 2019). More broadly, biases in the mod- els can comprise a wide range of harmful behaviors that may affect different social groups for various reasons (Blodgett et al., 2020). In this work, we take a different focus and study stereotypes that emerge within pretrained LMs in- stead. While bias is a personal preference that can be harmful when the tendency interferes with the ability to be impartial, stereotypes can be defined as a preconceived idea that (incorrectly) attributes general characteristics to all members of a group. While the two concepts are closely related i.e., stereotypes can evoke new biases or reinforce exist- ing ones, stereotypical thinking appears to be a cru- cial part of human cognition that often emerges im- plicitly (Hinton, 2017). Hinton (2017) argued that implicit stereotypical associations are established through Bayesian principles, where the experience of their prevalence in the world of the perceiver causes …

Content truncated to 500 words