HittER: Hierarchical Transformers for Knowledge Graph Embeddings

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10395–10407 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10395 HittER: Hierarchical Transformers for Knowledge Graph Embeddings Sanxing Chen∗ University of Virginia sc3hn@virginia.edu Xiaodong Liu, Jianfeng Gao Microsoft Research {xiaodl,jfgao}@microsoft.com Jian Jiao, Ruofei Zhang Microsoft Bing Ads {jiajia,bzhang}@microsoft.com Yangfeng Ji University of Virginia yangfeng@virginia.edu Abstract This paper examines the challenging problem of learning representations of entities and rela- tions in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical Transformer model to jointly learn Entity- relation composition and Relational contextu- alization based on a source entity’s neighbor- hood. Our proposed model consists of two dif- ferent Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational infor- mation from outputs of the bottom block. We further design a masked entity prediction task to balance information from the relational con- text and the source entity itself. Experimental results show that HittER achieves new state- of-the-art results on multiple link prediction datasets. We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets. 1 Introduction Knowledge graphs (KG) are a major form of knowl- edge bases where knowledge is stored as graph- structured data. Because of their broad applications in various intelligent systems including natural lan- guage understanding (Logan et al., 2019; Zhang et al., 2019b; Hayashi et al., 2020) and reason- ing (Riedel et al., 2013; Xiong et al., 2017; Bauer et al., 2018; Verga et al., 2021), learning represen- tations of knowledge graphs has been studied in a large body of literature. To learn high quality representations of knowl- edge graphs, many researchers adopt the idea of mapping the entities and relations in a knowledge graph to points in a vector space. These knowledge graph embedding (KGE) methods usually lever- age geometric properties in the vector space, such ∗Work was done during an internship at Microsoft Bing Ads. Sunnyvale California Cupertino United States adjoins state country Figure 1: An example subgraph sampled from FB15K- 237. Four nodes (entities) are connected by three different types of relations representing facts like Sunnyvale belongs to the state of California. as translation (Bordes et al., 2013), bilinear trans- formations (Yang et al., 2015, DistMult), or rota- tion (Sun et al., 2018). Multi-layer convolutional networks are also used for KGE (Dettmers et al., 2018, ConvE). Such KGE methods are conceptu- ally simple and can be applied to tasks like factoid question answering (Saxena et al., 2020) and lan- guage modeling (Peters et al., 2019). However, it is rather challenging to encode all of the information about an entity into a single vec- tor. For example, to infer the missing object in the incomplete triplet <Sunnyvale, county, ?> (Figure 1), traditional KGE methods rely on the geographic information stored in the embedding of Sunnyvale. While we can read such informa- tion from its graph context, …

Content truncated to 500 words