Efficient Hyper-parameter Search for Knowledge Graph Embedding

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2715 - 2735 May 22-27, 2022 c⃝2022 Association for Computational Linguistics KGTuner: Efficient Hyper-parameter Search for Knowledge Graph Learning Yongqi Zhang1 Zhanke Zhou2 Quanming Yao3 Yong Li3 14Paradigm Inc., Beijing, China 2Hong Kong Baptist University, Hong Kong, China 3Department of Electronic Engineering, Tsinghua University, Beijing, China zhangyongqi@4paradigm.com, cszkzhou@comp.hkbu.edu.hk, qyaoaa/liyong07@tsinghua.edu.cn Abstract While hyper-parameters (HPs) are important for knowledge graph (KG) learning, existing methods fail to search them efficiently. To solve this problem, we first analyze the proper- ties of different HPs and measure the trans- fer ability from small subgraph to the full graph. Based on the analysis, we propose an efficient two-stage search algorithm KG- Tuner, which efficiently explores HP config- urations on small subgraph at the first stage and transfers the top-performed configurations for fine-tuning on the large full graph at the second stage. Experiments show that our method can consistently find better HPs than the baseline algorithms within the same time budget, which achieves 9.1% average relative improvement for four embedding models on the large-scale KGs in open graph benchmark. Our code is released in https://github. com/AutoML-Research/KGTuner. 1 1 Introduction Knowledge graph (KG) is a special kind of graph structured data to represent knowledge through en- tities and relations between the entities (Wang et al., 2017; Ji et al., 2021). Learning from KG aims to discover the latent properties from KGs to infer the existence of interactions among entities or the types of entities (Wang et al., 2017; Zhang and Yao, 2022). KG embedding, which encodes enti- ties and relations as low dimensional vectors, is an important technique to learn from KGs (Wang et al., 2017; Ji et al., 2021). The existing models range from translational distance models (Bordes et al., 2013), tensor factorization models (Nickel et al., 2011; Trouillon et al., 2017; Balaževi´c et al., 2019), neural network models (Dettmers et al., 2017; Guo et al., 2019), to graph neural networks (Schlichtkrull et al., 2018; Vashishth et al., 2020). 1The work is performed when Z. Zhou was an intern in 4Paradigm, and correspondence is to Q. Yao. Hyper-parameter (HP) search (Claesen and De Moor, 2015) is very essential for KG learn- ing. In this work, we take KG embedding methods (Wang et al., 2017), as a good example to study the impact of HPs to KG learning. As studied, the HP configurations greatly influence the model perfor- mance (Ruffinelli et al., 2019; Ali et al., 2020). An improper HP configuration will impede the model from stable convergence, while an appropriate one can make considerable promotion to the model per- formance. Indeed, studying the HP configurations can help us make a more scientific understanding of the contributions made by existing works (Rossi et al., 2021; Sun et al., 2020). In addition, it is also important to search for an optimal HP configura- tion when adopting KG embedding methods to the real-world applications (Bordes et al., 2014; Zhang et al., 2016; Saxena …

Content truncated to 500 words