One Word, Two Sides: Traces of Stance in Contextualized Word Representations

Proceedings of the 29th International Conference on Computational Linguistics, pages 3950–3959 October 12–17, 2022. 3950 One Word, Two Sides: Traces of Stance in Contextualized Word Representations Aina Garí Soler, Matthieu Labeau, Chloé Clavel LTCI, Télécom-Paris, Institut Polytechnique de Paris, France {aina.garisoler,matthieu.labeau,chloe.clavel}@telecom-paris.fr Abstract The way we use words is influenced by our opinion. We investigate whether this is re- flected in contextualized word embeddings. For example, is the representation of “animal” different between people who would abolish zoos and those who would not? We explore this question from a Lexical Semantic Change standpoint. Our experiments with BERT em- beddings derived from datasets with stance annotations reveal small but significant differ- ences in word representations between oppos- ing stances. 1 Introduction Our opinions are reflected in the way we talk. Peo- ple with opposing stances on a particular topic may use different words when talking about it. For ex- ample, only people against the use of face masks during the COVID-19 pandemic would sometimes refer to them as “muzzles”. In this paper, however, we do not investigate what words are used by each side. Instead, we compare how speakers who dis- agree on a subject use the same words. Specifically, we want to know whether contextual models cap- ture a difference between the representation of a word (e.g., “mask”) when it is used by people who are in favor vs. against a certain target (e.g., the compulsory use of face masks). We address this question from the perspective of Lexical Semantic Change (LSC). Work on LSC typically tries to detect word meaning changes across two or more periods of time (Tahmasebi et al., 2021), but its techniques have also been em- ployed to identify synchronic differences in word usage, for instance across different ages, genders, professions (Gonen et al., 2020), domains (Yin et al., 2018; Schlechtweg et al., 2019), or cultures (Garimella et al., 2016). As opposed to related studies that investigate LSC between different view- points (Azarbonyad et al., 2017; Rodriguez et al., Figure 1: Example instances of “mask” from the Covid19 stance dataset (Glandt et al., 2021). We com- pare the within- and the between-stance usage similarity. 2021), our goal is not to explore the usage of spe- cific words, and we do not evaluate our method based on the ranking of words by meaning stability. We rather want to determine whether vector repre- sentations reflect a higher similarity in word usage within a stance than between different stances (see example in Figure 1). We explore this question relying on datasets annotated with stance informa- tion. Before that, we test different context-sensitive embedding models on a simulated scarce-data set- ting. This allows us to select a robust representation type that can identify the words that are used most differently between stances. Our long-term goal is to detect differences in word usage between speakers in a conversation, which could point to their level of conceptual align- ment (Stolk et al., 2016); that is, the extent to which dialog participants “mean the …

Content truncated to 500 words