Lexical Relation Mining in Neural Word Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 1299–1311 Barcelona, Spain (Online), December 8-13, 2020 1299 Lexical Relation Mining in Neural Word Embeddings Aishwarya Jadhav Heinz College Carnegie Mellon University, Pittsburgh, PA, USA ajadhav2@andrew.cmu.edu Yifat Amir Electrical Engineering and Computer Sciences University of California, Berkeley, CA, USA yifatamir@berkeley.edu Zachary A. Pardos Graduate School of Education University of California, Berkeley, CA, USA pardos@berkeley.edu Abstract Work with neural word embeddings and lexical relations has largely focused on confirmatory experiments which use human-curated examples of semantic and syntactic relations to validate against. In this paper, we explore the degree to which lexical relations, such as those found in popular validation sets, can be derived and extended from a variety of neural embeddings us- ing classical clustering methods. We show that the Word2Vec space of word-pairs (i.e., offset vectors) significantly outperforms other more contemporary methods, even in the presence of a large number of noisy offsets. Moreover, we show that via a simple nearest neighbor approach in the offset space, new examples of known relations can be discovered. Our results speak to the amenability of offset vectors from non-contextual neural embeddings to find semantically coher- ent clusters. This simple approach has implications for the exploration of emergent regularities and their examples, such as emerging trends on social media and their related posts. 1 Introduction Word vector models such as Word2Vec (Mikolov et al., 2013a), derived empirically from large cor- pora of natural language, provide the opportunity to explore what constitutes a linguistic regularity. Conventionally, lexical relations in word vector space have been defined by collections of relatively consistent relationships, or vector offsets, between word-pairs. The presence of these relationships has been established through confirmatory analysis (Levy and Goldberg, 2014), in which a pair of relation examples constructed from prior knowledge is validated to exist in the space by way of analogy (e.g. geese − goose + mouse ≈ mice) (Finley et al., 2017). The meaning of these collections, or the nature of the relationship between their respective pairs, can be characterized by relation type descriptions such as syntactic (e.g., “plurality”) and semantic (e.g., “capital-country”) relations. In this paper, we explore how well lexical relation examples can be clustered using word vectors extracted from state-of-the-art contextual and non-contextual neural word embedding methods. Further- more, we demonstrate a method for approximating the number of true lexical relations in a noisy offset space. Contextual embeddings such as BERT (Devlin et al., 2018) are the new gold standard on a va- riety of NLP tasks and have also been shown to possess relational and factual knowledge (Petroni et al., 2019). We perform the unsupervised task of clustering relation examples by using contextual word vectors from BERT; non-contextual word vectors from FastText (Bojanowski et al., 2017) Word2Vec skip-gram (Mikolov et al., 2013a); and explicitly modeled relation vectors (Camacho Collados et al., 2019). Our results show that Word2Vec offsets outperform relation vectors from other embeddings on this task. We further explore the amenability of the Word2vec offset space …

Content truncated to 500 words