Relational Word Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3286–3296 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3286 Relational Word Embeddings Jose Camacho-Collados Luis Espinosa-Anke School of Computer Science and InformaticsSteven Schockaert Cardiff University, United Kingdom {camachocolladosj,espinosa-ankel,schockaerts1}@cardiff.ac.uk Abstract While word embeddings have been shown to implicitly encode various forms of attribu- tional knowledge, the extent to which they capture relational information is far more lim- ited. In previous work, this limitation has been addressed by incorporating relational knowl- edge from external knowledge bases when learning the word embedding. Such strategies may not be optimal, however, as they are lim- ited by the coverage of available resources and conflate similarity with other forms of related- ness. As an alternative, in this paper we pro- pose to encode relational knowledge in a sep- arate word embedding, which is aimed to be complementary to a given standard word em- bedding. This relational word embedding is still learned from co-occurrence statistics, and can thus be used even when no external knowl- edge base is available. Our analysis shows that relational word vectors do indeed capture in- formation that is complementary to what is en- coded in standard word embeddings. 1 Introduction Word embeddings are paramount to the success of current natural language processing (NLP) meth- ods. Apart from the fact that they provide a con- venient mechanism for encoding textual informa- tion in neural network models, their importance mainly stems from the remarkable amount of lin- guistic and semantic information that they capture. For instance, the vector representation of the word Paris implicitly encodes that this word is a noun, and more specifically a capital city, and that it describes a location in France. This information arises because word embeddings are learned from co-occurrence counts, and properties such as be- ing a capital city are reflected in such statistics. However, the extent to which relational knowledge (e.g. Trump was the successor of Obama) can be learned in this way is limited. Previous work has addressed this by incorporat- ing external knowledge graphs (Xu et al., 2014; Celikyilmaz et al., 2015) or relations extracted from text (Chen et al., 2016). However, the suc- cess of such approaches depends on the amount of available relational knowledge. Moreover, they only consider well-defined discrete relation types (e.g. is the capital of, or is a part of), whereas the appeal of vector space representations largely comes from their ability to capture subtle aspects of meaning that go beyond what can be expressed symbolically. For instance, the relationship be- tween popcorn and cinema is intuitively clear, but it is more subtle than the assertion that “popcorn is located at cinema”, which is how ConceptNet (Speer et al., 2017), for example, encodes this re- lationship1. In fact, regardless of how a word embedding is learned, if its primary aim is to capture similarity, there are inherent limitations on the kinds of re- lations they can capture. For instance, such word embeddings …

Content truncated to 500 words