Understanding the Source of Semantic Regularities in Word Embeddings

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 119–131 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 119 Understanding the Source of Semantic Regularities in Word Embeddings Hsiao-Yu Chiang School of Information University of California, Berkeley, USA hsiaoyuchiang@berkeley.edu Jose Camacho-Collados School of Computer Science and Informatics Cardiff University, United Kingdom camachocolladosj@cardiff.ac.uk Zachary A. Pardos Graduate School of Education University of California, Berkeley, USA pardos@berkeley.edu Abstract Semantic relations are core to how humans un- derstand and express concepts in the real world using language. Recently, there has been a thread of research aimed at modeling these re- lations by learning vector representations from text corpora. Most of these approaches focus strictly on leveraging the co-occurrences of re- lationship word pairs within sentences. In this paper, we investigate the hypothesis that exam- ples of a lexical relation in a corpus are fun- damental to a neural word embedding’s abil- ity to complete analogies involving the rela- tion. Our experiments, in which we remove all known examples of a relation from train- ing corpora, show only marginal degradation in analogy completion performance involving the removed relation. This finding enhances our understanding of neural word embeddings, showing that co-occurrence information of a particular semantic relation is the not the main source of their structural regularity. 1 Introduction The representation of words has been a long- standing task in natural language processing (NLP). The main underlying principle is known for decades, as explained by Firth (1957). This prin- ciple was based on the idea that the meaning of a word can be understood by its surrounding com- pany (i.e., the words in its context). Most modern representation learning theory in NLP is based on this assumption, with vector representation being the most successful area to date (Turney and Pantel, 2010). More recently, low-dimensional word repre- sentations learned from text corpora using neural networks (i.e., word embeddings) have emerged (Mikolov et al., 2013a; Pennington et al., 2014; Bojanowski et al., 2017) stemming from cogni- tive frameworks based on distributed representa- tion (Hinton et al., 1986; Feldman and Ballard, 1982). Neural word embeddings have been proven to contain useful information about concepts and entities, and provide a generalization boost to many NLP applications (Goldberg, 2017). Surprisingly, these representations have also been shown to ex- hibit linear relationships between words in the vec- tor space, demonstrated by analogy. For example, Mikolov et al. (2013b) showed that a simple op- eration such as king-man+woman will result in a point near queen in the vector space1. These word analogies have been extensively investigated in the literature, aiming to shed light on this surprising property. However, while there has been a body of research seeking to understand how these analogies work (Arora et al., 2016; Gittens et al., 2017; Etha- yarajh et al., 2019; Allen and Hospedales, 2019), and noting issues about their methodology (Linzen, 2016; Gladkova et al., 2016; Nissim et al., 2020), there has not been a specific analysis on the source of statistical …

Content truncated to 500 words