Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1671–1682, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning Ekaterina Vylomova,1 Laura Rimell,2 Trevor Cohn,1 and Timothy Baldwin1 1Department of Computing and Information Systems, University of Melbourne 2Computer Laboratory, University of Cambridge evylomova@gmail.com laura.rimell@cl.cam.ac.uk {tcohn,tbaldwin}@unimelb.edu.au Abstract Recent work has shown that simple vector subtraction over word embeddings is surpris- ingly effective at capturing different lexical relations, despite lacking explicit supervision. Prior work has evaluated this intriguing result using a word analogy prediction formulation and hand-selected relations, but the generality of the finding over a broader range of lexical relation types and different learning settings has not been evaluated. In this paper, we carry out such an evaluation in two learning settings: (1) spectral clustering to induce word rela- tions, and (2) supervised learning to classify vector differences into relation types. We find that word embeddings capture a surprising amount of information, and that, under suit- able supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items. 1 Introduction Learning to identify lexical relations is a fundamen- tal task in natural language processing (“NLP”), and can contribute to many NLP applications including paraphrasing and generation, machine translation, and ontology building (Banko et al., 2007; Hen- drickx et al., 2010). Recently, attention has been focused on iden- tifying lexical relations using word embeddings, which are dense, low-dimensional vectors ob- tained either from a “predict-based” neural net- work trained to predict word contexts, or a “count- based” traditional distributional similarity method combined with dimensionality reduction. The skip- gram model of Mikolov et al. (2013a) and other similar language models have been shown to per- form well on an analogy completion task (Mikolov et al., 2013b; Mikolov et al., 2013c; Levy and Goldberg, 2014a), in the space of relational sim- ilarity prediction (Turney, 2006), where the task is to predict the missing word in analogies such as A:B :: C: –?–. A well-known example involves predicting the vector queen from the vector com- bination king − man + woman, where linear operations on word vectors appear to capture the lexical relation governing the analogy, in this case OPPOSITE-GENDER. The results extend to several semantic relations such as CAPITAL-OF (paris−france+poland ≈ warsaw) and mor- phosyntactic relations such as PLURALISATION (cars − car + apple ≈ apples). Remarkably, since the model is not trained for this task, the re- lational structure of the vector space appears to be an emergent property. The key operation in these models is vector dif- ference, or vector offset. For example, the paris − france vector appears to encode CAPITAL-OF, pre- sumably by cancelling out the features of paris that are France-specific, and retaining the features that distinguish a capital city (Levy and Goldberg, 2014a). The success of the simple offset method on analogy completion suggests that the difference vectors …

Content truncated to 500 words