Using Pairwise Occurrence Information to Improve Knowledge Graph Completion on Large-Scale Datasets

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3591–3596, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3591 Using Pairwise Occurrence Information to Improve Knowledge Graph Completion on Large-Scale Datasets Esma Balkır1,2*, Masha Naslidnyk2, Dave Palfrey2 and Arpit Mittal2 1University of Edinburgh, Scotland, UK 2Amazon Research, Cambridge, UK 1esma.balkir@ed.ac.uk 2{naslidny, dpalfrey, mitarpit}@amazon.co.uk Abstract Bilinear models such as DistMult and ComplEx are effective methods for knowl- edge graph (KG) completion. However, they require large batch sizes, which becomes a performance bottleneck when training on large scale datasets due to memory con- straints. In this paper we use occurrences of entity-relation pairs in the dataset to construct a joint learning model and to increase the quality of sampled negatives during training. We show on three standard datasets that when these two techniques are combined, they give a significant improvement in performance, especially when the batch size and the number of generated negative examples are low relative to the size of the dataset. We then apply our techniques to a dataset containing 2 million entities and demonstrate that our model outperforms the baseline by 2.8% absolute on hits@1. 1 Introduction A Knowledge Graph (KG) is a collection of facts which are stored as triples, e.g. Berlin is-capital-of Germany. Even though knowledge graphs are es- sential for various NLP tasks, open domain knowl- edge graphs have missing facts. To tackle this is- sue, there has recently been considerable interest in KG completion methods, where the goal is to rank correct triples above incorrect ones. Embedding methods such as DistMult (Yang et al., 2014) and ComplEx (Trouillon et al., 2016) are simple and effective methods for this task, but are known to be sensitive to hyperparameter and loss function choices (Kadlec et al., 2017; Lacroix et al., 2018). When paired with the right loss func- tion, these methods need large minibatches and a large number of corrupted triples per each positive triple during training to reach peak performance. *Work done while the author was an intern 1 This causes memory issues for KGs in the wild, which are several magnitudes bigger than the com- mon benchmarking datasets. To address the issue of scalability, we develop a framework that could be used with any bilin- ear KG embedding model. We name our model JoBi (Joint model with Biased negative sam- pling). Our framework uses occurrences of entity- relation pairs to overcome data sparsity, and to bias the model to score plausible triples higher. The framework trains a base model jointly with an auxiliary model that uses occurrences of pairs within a given triple in the data as labels. For ex- ample, the auxiliary model would receive the la- bel 1 for the triple (Berlin is-capital-of France) if the pairs (Berlin is-capital-of) and (is-capital-of France) are present in the training data, while the base model would receive the label 0. The intuition for using bigram occurrences is to capture …

Content truncated to 500 words