“You are grounded!”: Latent Name Artifacts in Pre-trained Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6850–6861, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6850 “You are grounded!”: Latent Name Artifacts in Pre-trained Language Models Vered Shwartz1,2, Rachel Rudinger1,2,3, and Oyvind Tafjord1 1Allen Institute for Artificial Intelligence 2Paul G. Allen School of Computer Science & Engineering, University of Washington 3University of Maryland, College Park, MD {vereds,oyvindt}@allenai.org, rudinger@umd.edu Abstract Pre-trained language models (LMs) may per- petuate biases originating in their training cor- pus to downstream models. We focus on ar- tifacts associated with the representation of given names (e.g., Donald), which, depending on the corpus, may be associated with spe- cific entities, as indicated by next token pre- diction (e.g., Trump). While helpful in some contexts, grounding happens also in under- specified or inappropriate contexts. For exam- ple, endings generated for ‘Donald is a’ sub- stantially differ from those of other names, and often have more-than-average negative senti- ment. We demonstrate the potential effect on downstream tasks with reading comprehen- sion probes where name perturbation changes the model answers. As a silver lining, our ex- periments suggest that additional pre-training on different corpora may mitigate this bias. 1 Introduction Pre-trained language models (LMs) have trans- formed the NLP landscape. State-of-the-art per- formance across tasks is achieved by fine-tuning the latest LM on task-specific data. LMs provide an effective way to represent contextual information, including lexical and syntactic knowledge as well as world knowledge (Petroni et al., 2019). LMs conflate generic facts (e.g.“the US has a president”) with grounded knowledge regarding specific entities and events (e.g.“the (current) pres- ident is a male”), occasionally leading to gender and racial biases (e.g.“women can’t be presidents”) (May et al., 2019; Sheng et al., 2019). In this work we focus on the representations of given names in pre-trained LMs (Table 1). Prior work showed that the representations of named entities incorporate sentiment (Prabhakaran et al., 2019), which is often transferable across entities via a shared given name (Field and Tsvetkov, 2019). Model Main Corpus Type Gen. Cls. BERT (Devlin et al., 2019) Wikipedia × ∨ RoBERTa (Liu et al., 2019) Web × ∨ GPT (Radford et al., 2018) Fiction ∨ × GPT2 (Radford et al., 2019) Web ∨ × XLNet (Yang et al., 2019) Web ∨ ∨ TransformerXL (Dai et al., 2019) Wikipedia ∨ × Table 1: Pre-trained LMs and whether they are typi- cally used for generation (Gen.) or classification (Cls.). In a series of experiments we show that, depending on the corpus, some names tend to be grounded to specific entities, even in generic contexts. The most striking effect is of politicians in GPT2. For example, the name Donald: 1) predicts Trump as the next token with high probability; 2) gener- ated endings of “Donald is a” are easily distinguish- able from any other given name; 3) their sentiment is substantially more negative; and 4) this bias can potentially perpetuate to downstream tasks. Although these results are expected, their extent is surprising. Biased name representations may …

Content truncated to 500 words