Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962–5971 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5962 Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling Robert L. Logan IV∗ Nelson F. Liu†§ Matthew E. Peters§ Matt Gardner§ Sameer Singh∗ ∗University of California, Irvine, CA, USA † University of Washington, Seattle, WA, USA § Allen Institute for Artificial Intelligence, Seattle, WA, USA {rlogan, sameer}@uci.edu, {mattg, matthewp}@allenai.org, nfliu@cs.washington.edu Abstract Modeling human language requires the ability to not only generate fluent text but also en- code factual knowledge. However, traditional language models are only capable of remem- bering facts seen at training time, and often have difficulty recalling them. To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. We also introduce the Linked WikiText- 2 dataset,1 a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 bench- mark (Merity et al., 2017). In experiments, we demonstrate that the KGLM achieves signifi- cantly better performance than a strong base- line language model. We additionally com- pare different language models’ ability to com- plete sentences requiring factual knowledge, and show that the KGLM outperforms even very large language models in generating facts. 1 Introduction For language models to generate plausible sen- tences, they must be both syntactically coherent as well as consistent with the world they describe. Al- though language models are quite skilled at generat- ing grammatical sentences, and previous work has shown that language models also possess some de- gree of common-sense reasoning and basic knowl- edge (Vinyals and Le, 2015; Serban et al., 2016; Trinh and Le, 2019), their ability to generate fac- tually correct text is quite limited. The clearest limitation of existing language models is that they, at best, can only memorize facts observed during 1https://rloganiv.github.io/linked-wikitext-2 [Super Mario Land] is a [1989] [side-scrolling] [platform video game] developed and published by [Nintendo] as a [launch title] for their [Game Boy] [handheld game console]. Date 21 April 1989 Q828322 platform game Q8093 Nintendo Q647249 Super Mario Land Q186437 Game Boy Q941818 handheld game console Q2281714 side-scrolling video game Q1425505 launch game Publication Date genre publisher platform manufacturer instance of Figure 1: Linked WikiText-2 Example. A localized knowledge graph containing facts that are (possibly) conveyed in the sentence above. The graph is built by it- eratively linking each detected entity to Wikidata, then adding any relations to previously mentioned entities. Note that not all entities are connected, potentially due to missing relations in Wikidata. training. For instance, when conditioned on the text at the top of Figure 1, an AWD-LSTM language model (Merity et al., 2018) trained on Wikitext-2 assigns higher probability to the word “PlaySta- tion” than “Game Boy”, …

Content truncated to 500 words