Why does PairDiff work? - A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection

Proceedings of the 27th International Conference on Computational Linguistics, pages 2493–2504 Santa Fe, New Mexico, USA, August 20-26, 2018. 2493 Why does PairDiff work? – A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection Huda Hakami The University of Liverpool Liverpool, UK h.a.hakami@liv.ac.uk Kohei Hayashi Preferred Networks Tokyo, Japan hayashi.kohei@gmail.com Danushka Bollegala The University of Liverpool Liverpool, UK danushka.bollegala@liv.ac.uk Abstract Representing the semantic relations that exist between two given words (or entities) is an im- portant first step in a wide-range of NLP applications such as analogical reasoning, knowledge base completion and relational information retrieval. A simple, yet surprisingly accurate method for representing a relation between two words is to compute the vector offset (PairDiff) between their corresponding word embeddings. Despite the empirical success, it remains unclear as to whether PairDiff is the best operator for obtaining a relational representation from word embed- dings. We conduct a theoretical analysis of generalised bilinear operators that can be used to measure the ℓ2 relational distance between two word-pairs. We show that, if the word embed- dings are standardised and uncorrelated, such an operator will be independent of bilinear terms, and can be simplified to a linear form, where PairDiff is a special case. For numerous word embedding types, we empirically verify the uncorrelation assumption, demonstrating the general applicability of our theoretical result. Moreover, we experimentally discover PairDiff from the bilinear relational compositional operator on several benchmark analogy datasets. 1 Introduction Different types of semantic relations exist between words such as HYPERNYMY between ostrich and bird, or ANTONYMY between hot and cold. If we consider entities1, we can observe even a richer diver- sity of relations such as FOUNDER-OF between Bill Gates and Microsoft, or CAPITAL-OF between Tokyo and Japan. Identifying the relations between words and entities is important for various Natural Language Processing (NLP) tasks such as automatic knowledge base completion (Socher et al., 2013), analogical reasoning (Turney and Littman, 2005; Bollegala et al., 2009) and relational information re- trieval (Duc et al., 2010). For example, to solve a word analogy problem of the form “a is to b as c is to ?”, the relationship between the two words in the pair (a, b) must be correctly identified in order to find candidates d that have similar relations with c. For example, given the query “Bill Gates is to Microsoft as Steve Jobs is to ?”, a relational search engine must retrieve Apple Inc. because the FOUNDER-OF relation exists between the first and the second entity pairs. Two main approaches for creating relation embeddings can be identified in the literature. In the first approach, from given corpora or knowledge bases, word and relation embeddings are jointly learnt such that some objective is optimised (Guo et al., 2016; Yang et al., 2015; Nickel et al., 2016; Bordes et al., 2013; Rocktäschel et al., 2016; Minervini et al., 2017; Trouillon et al., 2016). In this approach, word and relation embeddings are considered to be independent parameters that must be learnt by …

Content truncated to 500 words