Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8980–8994, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8980 Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning Tao Shen1∗, Yi Mao2, Pengcheng He2, Guodong Long1, Adam Trischler3andWeizhu Chen2 1Australian AI Institute, School of Computer Science, FEIT, University of Technology Sydney 2Microsoft Dynamics 365 AI 3Microsoft Research, Montr´eal tao.shen@student.uts.edu.au, guodong.long@uts.edu.au {maoyi,penhe,adtrisch,wzchen}@microsoft.com Abstract In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learn- ing over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribu- tion is an entity masking scheme that ex- ploits relational knowledge underlying the text. This is fulfilled by using a linked knowl- edge graph to select informative entities and then masking their mentions. In addition, we use knowledge graphs to obtain distrac- tors for the masked entities, and propose a novel distractor-suppressed ranking objective that is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learn- ing from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from con- catenated graph triples. Experiments show that our proposed model achieves improved perfor- mance on five benchmarks, including question answering and knowledge base completion. 1 Introduction Self-supervised pre-trained language models (LMs) like ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) learn powerful contextualized repre- sentations. With task-specific modules and fine- tuning, they have achieved state-of-the-art results on a wide range of natural language processing tasks. Nevertheless, open questions remain about what these models have learned and improvements can be made along several directions. One such direction is, when downstream task performance ∗ Work done while the author was an intern at Microsoft. Figure 1: Taxonomy of different approaches to integrat- ing pre-trained LMs with knowledge graphs. depends on relational knowledge – the kind mod- eled by knowledge graphs 1 (KGs) – directly fine- tuning a pre-trained LM often yields sub-optimal results, even though some works (Petroni et al., 2019; Davison et al., 2019) show pre-trained LMs have been partially equipped with such knowledge. To address this shortcoming, several recent works attempt to integrate KGs into pre-trained LMs. These approaches can be coarsely catego- rized into two classes, as shown in Figure 1. The first line of methods retrieves a KG subgraph (Liu et al., 2019a; Lin et al., 2019; Lv et al., 2019) and/or pre-trained graph embeddings (Zhang et al., 2019b; Peters et al., 2019) via entity linking during both training and inference on downstream tasks. While these methods inject domain-specific knowl- edge directly into language representations, they rely heavily on the performance of the linking al- gorithm and/or the quality of graph embeddings. Graph embeddings, to be tractable over large-scale KGs, are often learned using …

Content truncated to 500 words