Knowledge Base Embedding By Cooperative Knowledge Distillation

Proceedings of the 28th International Conference on Computational Linguistics, pages 5579–5590 Barcelona, Spain (Online), December 8-13, 2020 5579 Knowledge Base Embedding By Cooperative Knowledge Distillation Raphael Sourty1,2, Jose G. Moreno1, Francois-Paul Servant2, Lynda Tamine1 1Universit´e Paul Sabatier, IRIT, Toulouse, France 2Renault {raphael.sourty, jose.moreno, tamine}@irit.fr francois-paul.servant@renault.com Abstract Knowledge bases are increasingly exploited as gold standard data sources which benefit vari- ous knowledge-driven NLP tasks. In this paper, we explore a new research direction to perform knowledge base (KB) representation learning grounded with the recent theoretical framework of knowledge distillation over neural networks. Given a set of KBs, our proposed approach KD- MKB, learns KB embeddings by mutually and jointly distilling knowledge within a dynamic teacher-student setting. Experimental results on two standard datasets show that knowledge dis- tillation between KBs through entity and relation inference is actually observed. We also show that cooperative learning significantly outperforms the two proposed baselines, namely tradi- tional and sequential distillation. 1 Introduction Knowledge Bases (KB), organizing structured information about entities (nodes), and relations (edges) as graphs, are increasingly exploited as gold standard data sources for a broad range of human-AI tasks including language modeling (Logan et al., 2019), question answering (Shen et al., 2019) and semantic search (Bast et al., 2016). Although typical KBs may include a huge amount of observed knowledge through millions of entities and their relations, they are by nature incomplete since they can only capture a fraction of world knowledge. This limitation has given rise to extensive research work that focuses on the issue of predicting new knowledge from the observed one (Socher et al., 2013; Nickel et al., 2016). This issue has been successfully tackled by neural approaches for representation learning of KBs (Wang et al., 2017; Sun et al., 2019; Bordes et al., 2013; Yang et al., 2014). These models aim at representing KB entities and relations in low-dimensional embedding spaces, and supporting relational inferences using simple vector algebra. Recent years have witnessed increasing interest toward embedding models leveraged to connect multiple KBs (Liu et al., 2016; Chen et al., 2017; Trivedi et al., 2018; Zhu et al., 2017; Zhang et al., 2019). The key objective of multi-graph representation learning is to empower the entity and relation models with different graph contexts that potentially bridge different semantic contexts. To achieve this goal, embeddings are learned upon the combined triples across graphs. Although the above multi-graph rep- resentation learning methods have achieved promising results, they are still challenged by two main limitations. First, they are particularly suited to graph-alignment and machine translation as downstream tasks which necessarily leads to tackle computational challenges in large-scale KBs. Second, these meth- ods assume that each KB has access to all the entities and relations that are stored in the other KBs, while it may not be feasible neither relevant to the KBs to share unaligned information such as in personal KBs (Balog and Kenter, 2019). Following a different objective, we argue that apart from any downstream task, modeling the re- lational patterns across KBs …

Content truncated to 500 words