Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1924 - 1935 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Sense Embeddings are also Biased – Evaluating Social Biases in Static and Contextualised Sense Embeddings Yi Zhou1 Masahiro Kaneko2 University of Liverpool1, Tokyo Institute of Technology2, Amazon3 {y.zhou71,danushka}@liverpool.ac.uk masahiro.kaneko@nlp.c.titech.ac.jp Danushka Bollegala1,3∗ Abstract Sense embedding learning methods learn dif- ferent embeddings for the different senses of an ambiguous word. One sense of an ambigu- ous word might be socially biased while its other senses remain unbiased. In comparison to the numerous prior work evaluating the so- cial biases in pretrained word embeddings, the biases in sense embeddings have been relatively understudied. We create a benchmark dataset for evaluating the social biases in sense em- beddings and propose novel sense-specific bias evaluation measures. We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures. Our exper- imental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense- level, which are often ignored by the word-level bias evaluation measures.1 1 Introduction Sense embedding learning methods use different vectors to represent the different senses of an am- biguous word (Reisinger and Mooney, 2010; Nee- lakantan et al., 2014; Loureiro and Jorge, 2019). Although numerous prior works have studied so- cial biases in static and contextualised word embed- dings, social biases in sense embeddings remain un- derexplored (Kaneko and Bollegala, 2019, 2021a,a; Ravfogel et al., 2020; Dev et al., 2020; Schick et al., 2021; Wang et al., 2020). We follow Shah et al. (2020) and define social biases to be predictive biases with respect to pro- tected attributes made by NLP systems. Even if a word embedding is unbiased, some of its senses could still be associated with unfair social biases. ∗Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the Univer- sity of Liverpool and is not associated with Amazon. 1The dataset and evaluation scripts are available at github.com/LivNLP/bias-sense. Black people are friendly. Black people are unfriendly. Black dress is elegant. Black dress is ugly. race sense colour sense Figure 1: Example sentences from the Sense-Sensitive Social Bias dataset for the two senses of the ambigous word black. Top two sentences correspond to the colour sense of black, whereas the bottom two sentences cor- respond to its racial sense. Stereotypical examples that associate a sense with an unpleasant attribute are shown in red, whereas anti-stereotypical examples that asso- ciate a sense with a pleasant attribute are shown in blue. For example, consider the ambiguous word black, which has two adjectival senses according to the WordNet (Fellbaum and Miller, 1998): (1) black as a colour (being of the achromatic colour of maxi- mum darkness, sense-key=black%3:00:01) and (2) black as a race (of or belonging to a racial group especially …

Content truncated to 500 words