Inferring symmetry in natural language

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 259–266 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 259 Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models Zhiyuan Zhang1, Xiaoqian Liu1, 2, Yi Zhang1, Qi Su1, 2, Xu Sun1 and Bin He3 1 MOE Key Laboratory of Computational Linguistic, School of EECS, Peking University 2 School of Foreign Languages, Peking University 3 Huawei Noah’s Ark Lab {zzy1210,liuxiaoqian,zhangyi16,sukia,xusun}@pku.edu.cn hebin.nlp@huawei.com Abstract Conventional knowledge graph embedding (KGE) often suffers from limited knowledge representation, leading to performance degra- dation especially on the low-resource problem. To remedy this, we propose to enrich knowl- edge representation via pretrained language models by leveraging world knowledge from pretrained models. Specifically, we present a universal training framework named Pretrain- KGE consisting of three phases: semantic- based fine-tuning phase, knowledge extract- ing phase and KGE training phase. Extensive experiments show that our proposed Pretrain- KGE can improve results over KGE models, especially on solving the low-resource prob- lem. 1 Introduction Knowledge graphs (KGs) constitute an effective access to world knowledge for a wide variety of NLP tasks, such as entity linking (Luo et al., 2017), information retrieval (Xiong et al., 2017), ques- tion answering (Hao et al., 2017) and recommen- dation system (Zhang et al., 2016). A typical KG such as Freebase (Bollacker et al., 2008) and Word- Net (Miller, 1995), consists of a set of triplets in the form of (h, r, t) with the head entity h and the tail entity t as nodes and relation r as edges in the graph. A triplet represents the relation be- tween two entities, e.g., (Steve Jobs, founded, Ap- ple Inc.). To learn effective representation of en- tities and relations in the graph, knowledge graph embedding (KGE) models are one of prominent approaches (Bordes et al., 2013; Ji et al., 2015; Lin et al., 2015; Sun et al., 2019; Nickel et al., 2011; Yang et al., 2015; Kazemi and Poole, 2018; Trouil- lon et al., 2016; Zhang et al., 2019). However, traditional KGE models often suffer from limited knowledge representation due to the sparse and noisy dataset annotations. It leads to performance degradation, especially on the low- resource problem. To address this issue, we pro- pose to enrich knowledge representation via pre- trained language models (i.e., BERT (Devlin et al., 2019)) given a semantic description of entities and relations. We propose to incorporate world knowl- edge from BERT to the entity and the relation rep- resentation. Although simply fine-tuning BERT can enrich the knowledge representation, it suf- fers from learning inadequate structure informa- tion observed in training triplets, which we have demonstrated when we analyze the rationality of the KGE-training phase. We propose a model-agnostic training frame- work for learning knowledge graph embedding con- sisting of three phases: semantic-based fine-tuning phase, knowledge extracting phase and KGE train- ing phase (see Fig. 1). During the semantic-based fine-tuning phase, we learn knowledge representa- tion via BERT given the semantic description of entities and relations as the input sequence. In this …

Content truncated to 500 words