Relation Induction in Word Embeddings Revisited

Proceedings of the 27th International Conference on Computational Linguistics, pages 1627–1637 Santa Fe, New Mexico, USA, August 20-26, 2018. 1627 Relation Induction in Word Embeddings Revisited Zied Bouraoui CRIL CNRS UMR 8188 Artois University F-62300 Lens, France bouraoui@cril.fr Shoaib Jameel School of Computing Medway Campus University of Kent, UK M.S.Jameel@kent.ac.uk Steven Schockaert School of Computer Science and Informatics Cardiff University, UK schockaerts1@cardiff.ac.uk Abstract Given a set of instances of some relation, the relation induction task is to predict which other word pairs are likely to be related in the same way. While it is natural to use word embeddings for this task, standard approaches based on vector translations turn out to perform poorly. To address this issue, we propose two probabilistic relation induction models. The first model is based on translations, but uses Gaussians to explicitly model the variability of these translations and to encode soft constraints on the source and target words that may be chosen. In the second model, we use Bayesian linear regression to encode the assumption that there is a linear relation- ship between the vector representations of related words, which is considerably weaker than the assumption underlying translation based models. 1 Introduction It has been observed that many lexical relationships can be modelled as vector translations in a word embedding space (Mikolov et al., 2013b; Pennington et al., 2014). Even though this remarkable property is now well-established, and is commonly used as a basis for evaluating word embedding models, its potential for knowledge base completion has only been explored in a preliminary way. In this paper, we are particularly interested in the following relation induction task, which was proposed in (Vylomova et al., 2016): given a set {(s1, t1), ..., (sn, tn)} of word pairs that are related in a given way, identify new word pairs (s, t) that are likely to be related in the same way. We will refer to s and t as the source and target word respectively; we will write w for the vector representation of the word w. n1One natural approach is to model the considered relation using the average translation vector r = � i(ti − si), and to accept (s, t) as a likely instance if cos(s + r, t) is sufficiently high. While this approach was found in (Drozd et al., 2016) to work well for analogy completion, for relation induction it typically leads to too many false positives. This is illustrated in Table 1 for the case where r is constructed from the instances of the capital of relation of the BATS dataset1. As can be seen in the table, most of the top-ranked pairs are actually incorrect. In practice, this problem is further exacerbated by the dramatic class imbalance: for typical vocabulary sizes there are tens (or hundreds) of billions of incorrect pairs, compared to only a few hundred correct instances. While correct instances may, on average, get a higher score than incorrect instances, as a result of this imbalance there will still be …

Content truncated to 500 words