BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2281–2290 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2281 BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models Bin He1, Di Zhou1, Jinghui Xiao1, Xin Jiang1, Qun Liu1, Nicholas Jing Yuan2, Tong Xu3 1Huawei Noah’s Ark Lab 2Huawei Cloud & AI 3School of Computer Science, University of Science and Technology of China {hebin.nlp, zhoudi7, xiaojinghui4, jiang.xin, qun.liu, nicholas.yuan}@huawei.com, tongxu@ustc.edu.cn Abstract Complex node interactions are common in knowledge graphs (KGs), and these inter- actions can be considered as contextualized knowledge exists in the topological structure of KGs. Traditional knowledge representation learning (KRL) methods usually treat a single triple as a training unit, neglecting the usage of graph contextualized knowledge. To uti- lize these unexploited graph-level knowledge, we propose an approach to model subgraphs in a medical KG. Then, the learned knowl- edge is integrated with a pre-trained language model to do the knowledge generalization. Ex- perimental results demonstrate that our model achieves the state-of-the-art performance on several medical NLP tasks, and the improve- ment above MedERNIE indicates that graph contextualized knowledge is beneficial. 1 Introduction In 1954, Harris (1954) proposed a distributional hypothesis that words occur in the same contexts tend to have similar meanings. Firth (1957) ex- plained the context-dependent nature of meaning in linguistics by his famous quotation “you shall know a word by the company it keeps” . Although the above-mentioned distributional hypothesis is proposed for language models, if we look at the knowledge graph from the perspective of this hy- pothesis, we can find that similar hypothesis exists in knowledge graphs (KGs). We call it KG distri- butional hypothesis: you shall know an entity by the relationships it involves. Given this hypothesis, contextualized informa- tion in language models can be mapped to knowl- edge graphs, which we call “graph contextualized knowledge”. Figure 1 illustrates a knowledge sub- graph that includes several medical entities. In this figure, four incoming and four outgoing neighbor- ing nodes (hereinafter called “in-entity” and “out- entity”) of node “Bacterial pneumonia” are linked Figure 1: A subgraph extracted from a medical knowl- edge graph. The rectangles represent entities and di- rected arrows denote relations. by various relation paths. These linked nodes and correlations can be seen as “graph contextualized information” of entity node “Bacterial pneumo- nia”. In this study, we will explore how to integrate graph contextualized knowledge into pre-trained language models. Pre-trained language models learn contextual- ized word representations on large-scale text cor- pus through self-supervised learning methods, and obtain new state-of-the-art (SOTA) results on most downstream tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). This gradually becomes a new paradigm for natural language pro- cessing research. Recently, several knowledge- enhanced pre-trained language models have been proposed, such as ERNIE-Baidu (Sun et al., 2019), ERNIE-Tsinghua (Zhang et al., 2019a), WKLM (Xiong et al., 2019) and K-ADAPTER (Wang et al., 2020). In this study, since we need to learn graph contex- tualized knowledge in a large-scale …

Content truncated to 500 words