Word associations and the distance properties of context-aware word embeddings

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 376–385 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 376 Word associations and the distance properties of context-aware word embeddings Maria A. Rodriguez University of Geneva Maria.AnduezaRodriguez@unige.ch Paola Merlo University of Geneva Paola.Merlo@unige.ch Abstract What do people know when they know the meaning of words? Word associations have been widely used to tap into lexical repre- sentations and their structure, as a way of probing semantic knowledge in humans. We investigate whether current word embedding spaces (contextualized and uncontextualized) can be considered good models of human lexi- cal knowledge by studying whether they have comparable characteristics to human associa- tion spaces. We study the three properties of association rank, asymmetry of similarity and triangle inequality. We find that word embeddings are good mod- els of some word associations properties. They replicate well human associations between words, and, like humans, their context-aware variants show violations of the triangle in- equality. While they do show asymmetry of similarities, their asymmetries do not map those of human association norms. 1 Introduction What do people know when they know the mean- ing of words? Lexical semantic knowledge is rich and structured and comprises the knowledge of the relation between a word and its related concept, the relationships between concepts among themselves and between words themselves. Word associations —spontaneous elicitation of words by similarity, contrast or contiguity— have been widely used to tap into lexical representations and their structure, as a way of probing semantic knowledge in humans (De Deyne and Storms, 2014). To process language in a way that mirrors human expectations and to develop usable technology, we need computational representations of the meaning of words that correspond to speakers’ knowledge of words (De Deyne et al., 2016). Current word embeddings methods, without context or context- aware, represent lexical semantic knowledge as coordinates in a multi-dimensional space. Work in cognitive psychology, however, has argued that human lexical knowledge and in particular word associations are not well represented by geomet- rical models (Tversky, 1977; Gati and Tversky, 1982; Tversky and Hutchinson, 1986). More pre- cisely, similarity judgments of associations in hu- mans have been shown not to exhibit the properties of true distances. For example, associations are not symmetric: speakers will indicate that North Korea is more similar to China than China is to North Korea. In an analysis inspired by Tversky (1977)’s cri- tique of spatial measures of similarity in word associations, and by recent work on topic mod- els and word embeddings (Griffiths et al., 2007; Nematzadeh et al., 2017), we compare current word embedding spaces to the large human norm- ing study by Nelson et al. (2004) and investigate whether these word embedding spaces, especially those based on transformers, —the lexical represen- tations used by almost all our current architectures— share some properties with human word associ- ations, as human-like representations of lexical meaning. 2 Word Associations The task of word free association consists in provid- ing a cue …

Content truncated to 500 words