Lifted Rule Injection for Relation Embeddings

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1389–1399, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Lifted Rule Injection for Relation Embeddings Thomas Demeester Ghent University - iMinds Ghent, Belgium tdmeeste@intec.ugent.be Tim Rockt¨aschel and Sebastian Riedel University College London London, UK {t.rocktaschel,s.riedel}@cs.ucl.ac.uk Abstract Methods based on representation learning cur- rently hold the state-of-the-art in many natural language processing and knowledge base in- ference tasks. Yet, a major challenge is how to efficiently incorporate commonsense knowl- edge into such models. A recent approach reg- ularizes relation and entity representations by propositionalization of first-order logic rules. However, propositionalization does not scale beyond domains with only few entities and rules. In this paper we present a highly ef- ficient method for incorporating implication rules into distributed representations for au- tomated knowledge base construction. We map entity-tuple embeddings into an approxi- mately Boolean space and encourage a partial ordering over relation embeddings based on implication rules mined from WordNet. Sur- prisingly, we find that the strong restriction of the entity-tuple embedding space does not hurt the expressiveness of the model and even acts as a regularizer that improves general- ization. By incorporating few commonsense rules, we achieve an increase of 2 percentage points mean average precision over a matrix factorization baseline, while observing a neg- ligible increase in runtime. 1 Introduction Current successful methods for automated knowl- edge base construction tasks heavily rely on learned distributed vector representations (Nickel et al., 2012; Riedel et al., 2013; Socher et al., 2013; Chang et al., 2014; Neelakantan et al., 2015; Toutanova et al., 2015; Nickel et al., 2015; Verga et al., 2016; Verga and McCallum, 2016). Although these mod- els are able to learn robust representations from large amounts of data, they often lack common- sense knowledge. Such knowledge is rarely explic- itly stated in texts but can be found in resources like PPDB (Ganitkevitch et al., 2013) or WordNet (Miller, 1995). Combining neural methods with symbolic com- monsense knowledge, for instance in the form of implication rules, is in the focus of current research (Rockt¨aschel et al., 2014; Wang et al., 2014; Bow- man et al., 2015; Wang et al., 2015; Vendrov et al., 2016; Hu et al., 2016; Rockt¨aschel and Riedel, 2016; Cohen, 2016). A recent approach (Rockt¨aschel et al., 2015) regularizes entity-tuple and relation em- beddings via first-order logic rules. To this end, ev- ery first-order rule is propositionalized based on ob- served entity-tuples, and a differentiable loss term is added for every propositional rule. This approach does not scale beyond only a few entity-tuples and rules. For example, propositionalizing the rule ∀x : isMan(x) ⇒ isMortal(x) would result in a very large number of loss terms on a large database. In this paper, we present a method to incorporate simple rules while maintaining the computational efficiency of only modeling training facts. This is achieved by minimizing an upper bound of the loss that encourages the implication between rela- tions to hold, entirely independent from …

Content truncated to 500 words