Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2364–2375 June 6–11, 2021. ©2021 Association for Computational Linguistics 2364 Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis Xutan Peng Guanyi Chen Chenghua Lin ∗ Mark Stevenson Department of Computer Science, The University of Sheffield Department of Information and Computing Sciences, Utrecht University {x.peng, c.lin, mark.stevenson}@shef.ac.uk g.chen@uu.nl Abstract Knowledge Graph Embeddings (KGEs) have been intensively explored in recent years due to their promise for a wide range of applica- tions. However, existing studies focus on im- proving the final model performance without acknowledging the computational cost of the proposed approaches, in terms of execution time and environmental impact. This paper proposes a simple yet effective KGE frame- work which can reduce the training time and carbon footprint by orders of magnitudes com- pared with state-of-the-art approaches, while producing competitive performance. We high- light three technical innovations: full batch learning via relational matrices, closed-form Orthogonal Procrustes Analysis for KGEs, and non-negative-sampling training. In addition, as the first KGE method whose entity embed- dings also store full relation information, our trained models encode rich semantics and are highly interpretable. Comprehensive experi- ments and ablation studies involving 13 strong baselines and two standard datasets verify the effectiveness and efficiency of our algorithm. 1 Introduction The recent growth in energy requirements for Nat- ural Language Processing (NLP) algorithms has led to the recognition of the importance of com- putationally cheap and eco-friendly approaches (Strubell et al., 2019). The increase in computa- tional requirements can, to a large extent, be at- tributed to the popularity of massive pre-trained models, such as Language Models (e.g., BERT (De- vlin et al., 2019) and GPT-3 (Brown et al., 2020)) and Knowledge Graph Embeddings (KGEs, e.g., SACN (Shang et al., 2019)), that require significant resources to train. A number of solutions have been proposed such as reducing the number of parame- ters the model contains. For instance, Sanh et al. (2019) introduced a distilled version of BERT and ∗Chenghua Lin is the corresponding author. Zhang et al. (2019) decreased the parameters used for training KGEs with the help of the quaternion. In contrast with previous work, this paper explores algorithmic approaches to the development of effi- cient KGE techniques. Knowledge Graphs are core to many NLP tasks and downstream applications, such as question an- swering (Saxena et al., 2020), dialogue agents (He et al., 2017), search engines (Dong et al., 2014) and recommendation systems (Guo et al., 2020). Facts stored in a knowledge graph are always in the for- mat of tuples consisting of one head entity, one tail entity (both are nodes in knowledge graphs) and a relation (an edge in knowledge graphs) between them. KGEs learn representations of relations and entities in a knowledge graph, which are then utilised in downstream tasks like predicting miss- ing relations (Bordes et al., 2013; Sun et al., 2019; Tang et al., 2020). The application of deep learn- ing …

Content truncated to 500 words