ValNorm Quantifies Semantics to Reveal Consistent Valence Biases Across Languages and Over Centuries

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7203–7218 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7203 ValNorm Quantifies Semantics to Reveal Consistent Valence Biases Across Languages and Over Centuries Autumn Toney-Wails Georgetown University autumn.toney@georgetown.edu Aylin Caliskan University of Washington aylin@uw.edu Abstract Word embeddings learn implicit biases from linguistic regularities captured by word co- occurrence statistics. By extending methods that quantify human-like biases in word embed- dings, we introduce ValNorm, a novel intrinsic evaluation task and method to quantify the va- lence dimension of affect in human-rated word sets from social psychology. We apply Val- Norm on static word embeddings from seven languages (Chinese, English, German, Polish, Portuguese, Spanish, and Turkish) and from historical English text spanning 200 years. Val- Norm achieves consistently high accuracy in quantifying the valence of non-discriminatory, non-social group word sets. Specifically, Val- Norm achieves a Pearson correlation of ρ = 0.88 for human judgment scores of valence for 399 words collected to establish pleasantness norms in English. In contrast, we measure gen- der stereotypes using the same set of word em- beddings and find that social biases vary across languages. Our results indicate that valence associations of non-discriminatory, non-social group words represent widely-shared associa- tions, in seven languages and over 200 years. 1 Introduction New transparency-enhancing methods for static word embedding evaluation incorporate cross- disciplinary techniques that can quantify widely- accepted, intrinsic characteristics of words (Bakarov, 2018; Faruqui et al., 2016; Schnabel et al., 2015; Hollenstein et al., 2019). A promising approach for developing transparent intrinsic evalu- ation tasks is to evaluate word embeddings through the lens of cognitive lexical semantics, which cap- tures the social and psychological responses of hu- mans to words and language (Hollenstein et al., 2019; Osgood, 1964; Osgood et al., 1975). Such an approach could provide a representativeness evalu- ation method for word embeddings used in quanti- fying and studying biases. This paper presents the computational approach ValNorm, that accurately quantifies the valence di- mension of biases and affective meaning in word embeddings, to analyze widely shared associations of non-social group words. Implicit biases, as well as the intrinsic pleasantness or goodness of things, namely valence, have been well researched with human subjects (Greenwald et al., 1998; Rus- sell, 1983; Russell and Mehrabian, 1977). Valence is one of the principal dimensions of affect and cognitive heuristics that shape attitudes and bi- ases in humans (Harmon-Jones et al., 2013). Va- lence is described as the affective quality referring to the intrinsic attractiveness/goodness or averse- ness/badness of an event, object, or situation (Frijda et al., 1986; Osgood et al., 1957). For word em- beddings, we define valence bias as the semantic evaluation of pleasantness or unpleasantness that is associated with words (e.g., kindness is associated with pleasantness and torture is associated with unpleasantness). Word embedding evaluation tasks are methods to measure the quality and accuracy of learned word vector representations from a given text cor- pus. The two main types of evaluation tasks are intrinsic evaluation, …

Content truncated to 500 words