Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 1737–1743 Barcelona, Spain (Online), December 8-13, 2020 1737 Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models Bosung Kim1, Taesuk Hong2, Youngjoong Ko1, Jungyun Seo2 1Sungkyunkwan University, Suwon, Gyeonggi-do, Korea 2Sogang University, Seoul, Korea {bosungkim17,lino.taesuk}@gmail.com yjko@skku.edu, seojy@sogang.ac.kr Abstract As research on utilizing human knowledge in natural language processing has attracted consid- erable attention in recent years, knowledge graph (KG) completion has come into the spotlight. Recently, a new knowledge graph completion method using a pre-trained language model, such as KG-BERT, was presented and showed high performance. However, its scores in ranking met- rics such as Hits@k are still behind state-of-the-art models. We claim that there are two main reasons: 1) failure in sufficiently learning relational information in knowledge graphs, and 2) difficulty in picking out the correct answer from lexically similar candidates. In this paper, we propose an effective multi-task learning method to overcome the limitations of previous works. By combining relation prediction and relevance ranking tasks with our target link prediction, the proposed model can learn more relational properties in KGs and properly perform even when lexical similarity occurs. Experimental results show that we not only largely improve the ranking performances compared to KG-BERT but also achieve the state-of-the-art performances in Mean Rank and Hits@10 on the WN18RR dataset. 1 Introduction A Knowledge Graph (KG) is a graph-structured knowledge base, where real-world knowledge is rep- resented in the form of triple (h, r, t): (head entity, relation, tail entity) which means h and t have a relationship r. Entities and the relation in a triple are denoted as nodes and an edge of the graph, re- spectively. In recent years, Natural Language Processing (NLP) has benefited from utilizing KGs in various applications such as language modeling (Peters et al., 2019; Liu et al., 2019a), question answer- ing (Zhang et al., 2019; Huang et al., 2019), and machine reading (Yang and Mitchell, 2017). Since there has been an increasing demand for high-quality knowledge, the reliability of KG has also become important. Therefore, knowledge graph completion (a.k.a. link prediction), which identifies whether the triple in KG is valid or not, has been actively investigated. Several studies on the knowledge graph completion have been conducted (Bordes et al., 2013; Trouil- lon et al., 2016; Sun et al., 2019; Dettmers et al., 2018). They presented methods to model the connec- tivity patterns between entities in KG, and score functions to define the validity of the triple. However, these methods only consider graph structure and relational information depending on existing KG. Thus, they cannot predict well on triples that contain less frequent entities. Recently, addressing the sparseness problem of previous models, Yao et al. (2019) proposed a method called KG-BERT for knowledge graph completion, using entity descriptions and pre-trained language models. Even though KG-BERT signifi- cantly improved mean ranks using preliminary linguistic information from BERT (Devlin et al., 2018), the results in other ranking metrics such as MRR and Hit@k are still behind …

Content truncated to 500 words