Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 212–223 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 212 Dictionary-based Debiasing of Pre-trained Word Embeddings Masahiro Kaneko Tokyo Metropolitan University kaneko-masahiro@ed.tmu.ac.jp Danushka Bollegala∗ University of Liverpool, Amazon danushka@liverpool.ac.uk Abstract Word embeddings trained on large corpora have shown to encode high levels of unfair dis- criminatory gender, racial, religious and ethnic biases. In contrast, human-written dictionaries describe the meanings of words in a concise, objective and an unbiased manner. We pro- pose a method for debiasing pre-trained word embeddings using dictionaries, without requir- ing access to the original training resources or any knowledge regarding the word embedding algorithms used. Unlike prior work, our pro- posed method does not require the types of bi- ases to be pre-defined in the form of word lists, and learns the constraints that must be satis- fied by unbiased word embeddings automati- cally from dictionary definitions of the words. Specifically, we learn an encoder to generate a debiased version of an input word embed- ding such that it (a) retains the semantics of the pre-trained word embeddings, (b) agrees with the unbiased definition of the word according to the dictionary, and (c) remains orthogonal to the vector space spanned by any biased ba- sis vectors in the pre-trained word embedding space. Experimental results on standard bench- mark datasets show that the proposed method can accurately remove unfair biases encoded in pre-trained word embeddings, while pre- serving useful semantics. 1 Introduction Despite pre-trained word embeddings are useful due to their low dimensionality, memory and com- pute efficiency, they have shown to encode not only the semantics of words but also unfair discrimi- natory biases such as gender, racial or religious biases (Bolukbasi et al., 2016; Zhao et al., 2018a; Rudinger et al., 2018; Zhao et al., 2018b; Elazar ∗Danushka Bollegala holds concurrent appointments as a Professor at University of Liverpool and as an Amazon Scholar. This paper describes work performed at the Univer- sity of Liverpool and is not associated with Amazon. and Goldberg, 2018; Kaneko and Bollegala, 2019). On the other hand, human-written dictionaries act as an impartial, objective and unbiased source of word meaning. Although methods that learn word embeddings by purely using dictionaries have been proposed (Tissier et al., 2017), they have cover- age and data sparseness related issues because pre- compiled dictionaries do not capture the meanings of neologisms or provide numerous contexts as in a corpus. Consequently, prior work has shown that word embeddings learnt from large text corpora to outperform those created from dictionaries in downstream NLP tasks (Alsuhaibani et al., 2019; Bollegala et al., 2016). We must overcome several challenges when us- ing dictionaries to debias pre-trained word embed- dings. First, not all words in the embeddings will appear in the given dictionary. Dictionaries often have limited coverage and will not cover neolo- gisms, orthographic variants of words etc. that are likely to appear in large corpora. A lexicalised debiasing method …

Content truncated to 500 words