Discovering Differences in the Representation of People using Contextualized Semantic Axes

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3477 - 3494 December 7-11, 2022 ©2022 Association for Computational Linguistics Discovering Differences in the Representation of People using Contextualized Semantic Axes Li Lucy, Divya Tadimeti, and David Bamman University of California, Berkeley {lucy3_li, dtadimeti, dbamman}@berkeley.edu Abstract A common paradigm for identifying seman- tic differences across social and temporal con- texts is the use of static word embeddings and their distances. In particular, past work has compared embeddings against “semantic axes” that represent two opposing concepts. We ex- tend this paradigm to BERT embeddings, and construct contextualized axes that mitigate the pitfall where antonyms have neighboring repre- sentations. We validate and demonstrate these axes on two people-centric datasets: occupa- tions from Wikipedia, and multi-platform dis- cussions in extremist, men’s communities over fourteen years. In both studies, contextual- ized semantic axes can characterize differences among instances of the same word type. In the latter study, we show that references to women and the contexts around them have become more detestable over time. 1 Introduction Warning: This paper contains content that may be offensive or upsetting. Quantifying and describing the nature of lan- guage differences is key to measuring the impact of social and cultural factors on text. Past work has compared English embeddings for people to adjec- tives or concepts (Garg et al., 2018; Mendelsohn et al., 2020; Charlesworth et al., 2022), or projected embeddings against axes representing contrasting attributes (Turney and Littman, 2003; An et al., 2018; Kozlowski et al., 2019; Field and Tsvetkov, 2019; Mathew et al., 2020; Kwak et al., 2021; Lucy and Bamman, 2021b; Fraser et al., 2021; Grand et al., 2022). Static representations for the same word can also be juxtaposed across corpora that reflect different time periods (Gonen et al., 2020; Hamilton et al., 2016). This paradigm of using em- bedding distances to uncover socially meaningful patterns has also transferred over to studies that measure biases in contextualized embeddings, such as Wolfe and Caliskan (2021)’s finding that BERT beautiful ugly … gorgeous equipage, ornately attired … … a grotesque, monster-like costume… … there are beautiful women who are willing … … literally filled with garbage women … Figure 1: An axis is constructed using embeddings of adjectives in selected contexts. These contexts are pre- dictive of synonyms, but not antonyms, of the target adjective during masked language modeling. Token- level embeddings for people are then projected onto this axis. embeddings of less frequent minority names are closer to words related to unpleasantness. The use of “semantic axes” is enticing in that it offers an interpretable measurement of word differ- ences beyond a single similarity value (Turney and Littman, 2003; An et al., 2018; Kozlowski et al., 2019; Kwak et al., 2021). Words are projected onto axes where the poles represent antonymous con- cepts (such as beautiful–ugly), and the projected embedding’s location along the axis indicates how similar it is to either concept. Semantic axes con- structed using static, type-based embeddings have been …

Content truncated to 500 words