SeVeN: Augmenting Word Embeddings with Unsupervised Relation Vectors

Proceedings of the 27th International Conference on Computational Linguistics, pages 2653–2665 Santa Fe, New Mexico, USA, August 20-26, 2018. 2653 SeVeN: Augmenting Word Embeddings with Unsupervised Relation Vectors Luis Espinosa-Anke and Steven Schockaert School of Computer Science and Informatics, Cardiff University, UK {Espinosa-AnkeL,SchockaertS1}@cardiff.ac.uk Abstract We present SeVeN (Semantic Vector Networks), a hybrid resource that encodes relationships between words in the form of a graph. Different from traditional semantic networks, these re- lations are represented as vectors in a continuous vector space. We propose a simple pipeline for learning such relation vectors, which is based on word vector averaging in combination with an ad hoc autoencoder. We show that by explicitly encoding relational information in a ded- icated vector space we can capture aspects of word meaning that are complementary to what is captured by word embeddings. For example, by examining clusters of relation vectors, we observe that relational similarities can be identified at a more abstract level than with tradi- tional word vector differences. Finally, we test the effectiveness of semantic vector networks in two tasks: measuring word similarity and neural text categorization. SeVeN is available at bitbucket.org/luisespinosa/seven. 1 Introduction Word embedding models such as Skip-gram (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014) use fixed-dimensional vectors to represent the meaning of words. These word vectors essentially cap- ture a kind of similarity structure, which has proven to be useful in a wide range of Natural Language Processing (NLP) tasks. Today, one of the major applications of word embeddings is their interaction with neural network architectures, enabling a kind of generalization beyond those words that were only observed during training. For example, if a classification model has learned that news stories containing words such as ‘cinema’, ‘restaurant’ and ‘zoo’ tend to be categorized as ‘entertainment’, it may predict this latter label also for stories about theme parks due to the shared semantic properties encoded in word vectors. Word embeddings thus endow neural models with some form of world knowledge, without which they would be far less effective. This has prompted a prolific line of research focused on im- proving word embeddings not only with algorithmic sophistication, but also via explicit incorporation of external knowledge sources such as WordNet (Faruqui et al., 2015), BabelNet (Camacho-Collados et al., 2016) or ConceptNet (Speer et al., 2016). Regardless of how word vectors are learned, however, the use of fixed-dimensional representations constrains the kind of knowledge they can encode. Essentially, we can think of a word vector as a compact encoding of the salient attributes of the given word. For instance, the vector representation of lion might implicitly encode that this word is a noun, and that lions have attributes such as ‘dangerous’, ‘predator’ and ‘carnivorous’. Beyond these properties, word embeddings can also encode relational knowledge. For instance, the embedding might tell us that the words ‘lion’ and ‘zebra’ are semantically related, which together with the attributional knowledge that lions are predators and zebras are prey may allow us to plausibly infer that …

Content truncated to 500 words