Enhancing Contextual Word Representations Using Embedding of Neighboring Entities in Knowledge Graphs

Proceedings of the 29th International Conference on Computational Linguistics, pages 3175–3186 October 12–17, 2022. 3175 Enhancing Contextual Word Representations Using Embedding of Neighboring Entities in Knowledge Graphs Ryoko Tokuhisa Keisuke Kawano Akihiro Nakamura Satoshi Koide Toyota Central R&D Labs., Inc. {tokuhisa, kawano, a-nakamura, koide}@mosk.tytlabs.co.jp Abstract Pre-trained language models (PLMs) such as BERT and RoBERTa have dramatically im- proved the performance of various natural lan- guage processing tasks. Although these models are trained on large amounts of raw text, they have no explicit grounding in real-world enti- ties. Knowledge graphs (KGs) are manually annotated with factual knowledge and store the relations between nodes corresponding to en- tities as labeled edges. This paper proposes a mechanism called KG-attention, which inte- grates the structure of a KG into recent PLM architectures. Unlike the existing PLM+KG integration methods, KG-attention generalizes the embeddings of neighboring entities using the relation embeddings; accordingly, it can handle relations between unconnected entities in the KG. Experimental results demonstrated that our method achieved significant improve- ments in a relation classification task, an entity typing task, and several language comprehen- sion tasks. 1 Introduction Pre-trained language models (PLMs) have signifi- cantly improved the performance of various natural language processing (NLP) tasks (Devlin et al., 2019; Liu et al., 2019). Although these models are trained on large amounts of raw text, they have no explicit grounding in real-world entities. Figure 1 shows a causal reasoning task on which PLM fails. The performance can be improved by represent- ing factual knowledge as various relations between entities, e.g., “engine is part of a car”, in the PLM. Knowledge graphs (KGs) are manually anno- tated with factual knowledge and store the relations between the nodes corresponding to entities as la- beled edges (e.g., HasA, IsA, and PartOf) (Miller, 1995; Bollacker et al., 2008; Speer et al., 2016). Although the various relations recorded in KGs can potentially improve the performance of PLMs, KGs are graphs and structurally unsuitable for di- Figure 1: Example of the outputs of Choice of Plausible Alternatives (COPA), a well-known causal reasoning task. “PLM” represents the causal reasoning model us- ing BERTLARGE and “PLM+KG” represents the model using BERTLARGE and Knowledge Graph (WordNet). Solid and dotted lines indicate the relations between en- tities connected in KG and entities not connected in KG, respectively. The relation between unconnected entities were estimated from KG embeddings in our proposed method. rect incorporation into PLMs. Some of the more suitable forms of factual knowledge are pre-trained KG embeddings (Bordes et al., 2013; Sun et al., 2019; Nguyen, 2020; Chao et al., 2021). KG em- beddings embed two entities registered in a KG such that the corresponding embedding vectors are at a specific relative position depending on the rela- tion between the entities. The embedding vectors corresponding to the entities are called KG entity embeddings and the specific relative positions are called relation embeddings. In addition to avoid- ing the direct use of structural encoding of KGs, KG embeddings can be generalized; that is, entities having a …

Content truncated to 500 words