Knowledge Enhanced Contextual Word Representations

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 43–54, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 43 Knowledge Enhanced Contextual Word Representations Matthew E. Peters1, Mark Neumann1, Robert L. Logan IV2, Roy Schwartz1,3, Vidur Joshi1, Sameer Singh2, and Noah A. Smith1,3 1Allen Institute for Artificial Intelligence, Seattle, WA, USA 2University of California, Irvine, CA, USA 3Paul G. Allen School of Computer Science & Engineering, University of Washington {matthewp,markn,roys,noah}@allenai.org {rlogan,sameer}@uci.edu Abstract Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities. We propose a general method to embed multiple knowledge bases (KBs) into large scale models, and thereby enhance their representations with structured, human-curated knowledge. For each KB, we first use an integrated entity linker to retrieve relevant entity embeddings, then update con- textual word representations via a form of word-to-entity attention. In contrast to pre- vious approaches, the entity linkers and self- supervised language modeling objective are jointly trained end-to-end in a multitask set- ting that combines a small amount of entity linking supervision with a large amount of raw text. After integrating WordNet and a subset of Wikipedia into BERT, the knowledge en- hanced BERT (KnowBert) demonstrates im- proved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, en- tity typing, and word sense disambiguation. KnowBert’s runtime is comparable to BERT’s and it scales to large KBs. 1 Introduction Large pretrained models such as ELMo (Peters et al., 2018), GPT (Radford et al., 2018), and BERT (Devlin et al., 2019) have significantly im- proved the state of the art for a wide range of NLP tasks. These models are trained on large amounts of raw text using self-supervised objectives. How- ever, they do not contain any explicit grounding to real world entities and as a result have difficulty recovering factual knowledge (Logan et al., 2019). Knowledge bases (KBs) provide a rich source of high quality, human-curated knowledge that can be used to ground these models. In addition, they often include complementary information to that found in raw text, and can encode factual knowl- edge that is difficult to learn from selectional pref- erences either due to infrequent mentions of com- monsense knowledge or long range dependencies. We present a general method to insert multiple KBs into a large pretrained model with a Knowl- edge Attention and Recontextualization (KAR) mechanism. The key idea is to explicitly model entity spans in the input text and use an entity linker to retrieve relevant entity embeddings from a KB to form knowledge enhanced entity-span representations. Then, the model recontextual- izes the entity-span representations with word-to- entity attention to allow long range interactions between contextual word representations and all entity spans in the context. The entire KAR is in- serted between two layers in the middle of a …

Content truncated to 500 words