Knowledge Graph Embedding Compression

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2681–2691 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2681 Knowledge Graph Embedding Compression Mrinmaya Sachan Toyota Technological Institute at Chicago mrinmaya@ttic.edu Abstract Knowledge graph (KG) representation learn- ing techniques that learn continuous embed- dings of entities and relations in the KG have become popular in many AI applications. With a large KG, the embeddings consume a large amount of storage and memory. This is prob- lematic and prohibits the deployment of these techniques in many real world settings. Thus, we propose an approach that compresses the KG embedding layer by representing each en- tity in the KG as a vector of discrete codes and then composes the embeddings from these codes. The approach can be trained end-to- end with simple modifications to any existing KG embedding technique. We evaluate the approach on various standard KG embedding evaluations and show that it achieves 50-1000x compression of embeddings with a minor loss in performance. The compressed embeddings also retain the ability to perform various rea- soning tasks such as KG inference. 1 Introduction Knowledge graphs (KGs) are a popular way of stor- ing world knowledge, lending support to a number of AI applications such as search (Singhal, 2012), question answering (Lopez et al., 2013; Berant et al., 2013) and dialog systems (He et al., 2017; Young et al., 2018). Typical KGs are huge, consist- ing of millions of entities and relations. With the growth in use of KGs, researchers have explored ways to learn better representations of KGs in order to improve generalization and robust- ness in downstream tasks. In particular, there has been interest in learning embeddings of KGs in con- tinuous vector spaces (Bordes et al., 2011, 2013; Socher et al., 2013). KG embedding approaches represent entities as learnable continuous vectors while each relation is modeled as an operation in the same space such as translation, projection, etc. (Bordes et al., 2013; Wang et al., 2014; Lin et al., 2015; Ji et al., 2015). These approaches give us a way to perform reasoning in KGs with simple numerical computation in continuous spaces. Despite the simplicity and wide-applicability of KG embedding approaches, they have a few key issues. A major issue is that the number of embed- ding parameters grow linearly with the number of entities. This is challenging when we have millions or billions of entities in the KG, especially when there are a lot of sparse entities or relations in the KG. There is a clear redundancy in the continuous parameterization of embeddings given that many entities are actually similar to each other. This over- parameterization can lead to a drop in performance due to overfitting in downstream models. The large memory requirement of continuous representations also prevents models that rely on them from being deployed on modest user-facing computing devices such as mobile phones. To address this issue, we propose a coding scheme that replaces the traditional KG embedding layer by representing …

Content truncated to 500 words