A Mixture-of-Experts Model for Learning Multi-Facet Entity Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 508–518 Barcelona, Spain (Online), December 8-13, 2020 508 AprilE: Attention with Pseudo Residual Connection for Knowledge Graph Embedding Yuzhang Liu†, Peng Wang†,‡,≀∗, Yingtai Li‡, Yizhan Shao‡, Zhongkai Xu≀ †School of Computer Science and Engineering, Southeast University, China ‡School of Artificial Intelligence, Southeast University, China ≀School of Cyber Science and Engineering, Southeast University, China {yuzhangliu, pwang, lytai, syzmaxwell, xuzhongkai }@seu.edu.cn Abstract Knowledge graph embedding maps entities and relations into low-dimensional vector space. However, it is still challenging for many existing methods to model diverse relational patterns, es- pecially symmetric and antisymmetric relations. To address this issue, we propose a novel model, AprilE, which employs triple-level self-attention and pseudo residual connection to model rela- tional patterns. The triple-level self-attention treats head entity, relation, and tail entity as a sequence and captures the dependency within a triple. At the same time the pseudo residual connection retains primitive semantic features. Furthermore, to deal with symmetric and anti- symmetric relations, two schemas of score function are designed via a position-adaptive mecha- nism. Experimental results on public datasets demonstrate that our model can produce expressive knowledge embedding and significantly outperforms most of the state-of-the-art works. 1 Introduction Large scale knowledge graphs such as DBpedia (Auer et al., 2007), Freebase (Bollacker et al., 2008), and YAGO (Suchanek et al., 2007), have been shown useful to many applications including natural language understanding (Wang et al., 2017), question answering (Mohammed et al., 2018), and recommender systems (Wang et al., 2019). Knowledge graphs (KGs) (Dong et al., 2014; Hogan et al., 2020) are multi- relational graphs containing much factual information in the form of a triple (h, r, t), where h represents the head entity, t represents the tail entity, and r represents the relationship between h and t. Although the symbolic form of triples effectively represents structured data, it makes KGs hard to calculate se- mantic similarity between entities and relations. In this paper, we focus specifically on knowledge graph embedding, which maps entities and relations into low-dimensional vector space while preserving the inherent structures of entities and relations. There are diverse relations in KGs. Figure 1 shows symmetric and antisymmetric relational patterns in a KG. Barack Obama and Michelle Obama are in a marriage relation, and vice versa, therefore, marriage is a symmetric relation. For another case, Barack Obama Sr. is the father of Barack Obama but Barack Obama is not the father of Barack Obama Sr., thus father of is an antisymmetric relation. In addition, there are also different relational categories in Figure 1, which are one-to-one, one-to-many, many-to- one, and many-to-many. For example, both Barack Obama and Michelle Obama are doctorate of J.D. (Juris Doctor), which belongs to many-to-one relational category. The alma mater of Barack Obama are Columbia University and Harvard University, which belongs to one-to-many relational category. Many existing models are effective to process different relational categories. TransE (Bordes et al., 2013) interprets the relation as a translation operation on entities in the low-dimensional …

Content truncated to 500 words