A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 1292–1298 Barcelona, Spain (Online), December 8-13, 2020 1292 A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings Sapan Shah1,2, Sreedhar Reddy1, and Pushpak Bhattacharyya2 1TCS Research, Tata Consultancy Services, Pune 2Indian Institute of Technology Bombay, Mumbai {sapan.hs,sreedhar.reddy}@tcs.com pb@cse.iitb.ac.in Abstract We present a novel retrofitting model that can leverage relational knowledge available in a knowledge resource to improve word embeddings. The knowledge is captured in terms of relation inequality constraints that compare similarity of related and unrelated entities in the context of an anchor entity. These constraints are used as training data to learn a non-linear transformation function that maps original word vectors to a vector space respecting these constraints. The transformation function is learned in a similarity metric learning setting using Triplet network architecture. We applied our model to synonymy, antonymy and hypernymy relations in WordNet and observed large gains in performance over original distributional models as well as other retrofitting approaches on word similarity task and significant overall improvement on lexical entailment detection task. 1 Introduction Word embedding models (Pennington et al., 2014; Mikolov et al., 2013) are primarily inspired from the distributional hypothesis (Harris, 1954) viz. words that appear in similar context tend to have similar meaning. However, these models have one major drawback: they mix semantic similarity with other types of semantic relatedness (Hill et al., 2015). Consider for example, cheap and expensive. Though completely opposite in meaning, these words tend to occur in nearly identical contexts and end up having similar distributional vectors. This is problematic for many end applications such as sentiment analysis, text simplification, and so on. To address this issue, researchers have proposed various models to combine information from external knowledge sources such as WordNet, Freebase, etc. into unsupervised learning of word embeddings. These models mainly focus on the constraints extracted from various types of relations such as synonymy, antonymy, hypernymy, etc. At a high level, these models are categorized into: Joint specialization models (Yu and Dredze, 2014; Liu et al., 2015; Xu et al., 2014); and Retrofitting models (Faruqui et al., 2015; Wieting et al., 2015; Glavaˇs and Vuli´c, 2018; Kamath et al., 2019). Joint specialization models typically modify the optimization objective of distributional models by integrating the constraints into the objective function. Whereas, retrofitting models update the word vectors of distributional models in a post-processing training step using data generated from the constraints. Current retrofitting models have one limitation. They use constraints that tend to push cosine similarity to extremes (+1 or -1). While this works well for relations such as synonymy and anotonymy, it does not work so well for relations such as hypernymy, holonymy, etc. We need an approach that works for all relations while striking the right balance with distributional semantics. In this work, we present a new method to obtain constraints from all types of relations present in a knowledge resource. The constraints are in the form of relation inequalities. The central idea is: if an …

Content truncated to 500 words