Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers

Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers Andrew Silva∗ Pradyumna Tambwekar∗ School of Interactive Computing Georgia Institute of Technology Atlanta, GA {andrew.silva, ptambwekar3, matthew.gombolay}@gatech.edu Matthew Gombolay Abstract The ease of access to pre-trained transformers has enabled developers to leverage large-scale language models to build exciting applications for their users. While such pre-trained models offer convenient starting points for researchers and developers, there is little consideration for the societal biases captured within these model risking perpetuation of racial, gender, and other harmful biases when these models are deployed at scale. In this paper, we inves- tigate gender and racial bias across ubiquitous pre-trained language models, including GPT-2, XLNet, BERT, RoBERTa, ALBERT and Dis- tilBERT. We evaluate bias within pre-trained transformers using three metrics: WEAT, se- quence likelihood, and pronoun ranking. We conclude with an experiment demonstrating the ineffectiveness of word-embedding tech- niques, such as WEAT, signaling the need for more robust bias testing in transformers. 1 Introduction Transformer models represent the state-of-the-art for many natural language processing (NLP) tasks, such as question-answering (Devlin et al., 2019), dialogue (Smith et al., 2020), search results (Nayak, 2019), and more. Popular pre-trained models, such as those available from Hugging Face (Wolf et al., 2019), allow developers without extensive compu- tation power to benefit from these models. How- ever, it is important to fully understand the latent societal biases within these black-box transformer models. Without appropriately considering inher- ent biases, development on top of pre-trained trans- formers risks exacerbating and propagating racial, gender, and other biases writ large. Before transformers, word embedding models such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) were shown to ex- hibit systematic sexist (Bolukbasi et al., 2016) and ∗ These authors contributed equally to this work racist (Manzini et al., 2019) biases. Initial investi- gations into bias for transformers (Vig et al., 2020; Basta et al., 2019; Bommasani et al., 2020) have found that these new language models are similarly biased. As transformers are increasingly common- place, a more complete view of the inequalities, biases, or under-representations within pre-trained transformers becomes increasingly important. Yet, discovering bias in transformer models has proven to be more nuanced than bias-discovery in word embedding models (Kurita et al., 2019; May et al., 2019). Prior work on bias in modern trans- former models has used only a single test or metric at a time, which we show in this paper provides an incomplete view of the problem. Furthermore, we find evidence that certain tests are ill-suited to understanding bias in transformer architectures, supported by prior work (Blodgett et al., 2020). Moreover, we show that employing multiple tests is necessary for a full picture of the issue as no single test is currently sufficient. In the context of our work, “bias” refers specifi- cally to the preference of a model for one gender or race in the presence of an otherwise neutral context. As an example, consider the sequence “[MASK] wept upon arriving to the …

Content truncated to 500 words