Relational Memory-Augmented Language Models

Relational Memory-Augmented Language Models Qi Liu2∗, Dani Yogatama1, and Phil Blunsom1,2 1DeepMind, United Kingdom, 2University of Oxford, United Kingdom {qi.liu,phil.blunsom}@cs.ox.ac.uk dyogatama@deepmind.com Abstract We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and re- trieve relevant relations for a given context to improve text generation. Experiments on WikiText-103, WMT19, and enwik8 English datasets demonstrate that our approach pro- duces a better language model in terms of per- plexity and bits per character. We also show that relational memory improves coherence, is complementary to token-based memory, and enables causal interventions. Our model pro- vides a simple yet effective way to combine an autoregressive language model and a knowl- edge graph for more coherent and logical generation. 1 Introduction A core function of language is to communicate propositions (e.g., who did what to whom). As such, language models need to be able to generate this information reliably and coherently. Existing language models (Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020) do not have ex- plicit representations for such information and rely on it being implicitly encoded in their param- eters (Liu et al., 2019; Petroni et al., 2019; Wang et al., 2020). This encoding mechanism makes it difficult to interpret what the language mod- els know and often leads to generating illogical and contradictory contents. For example, Logan et al. (2019) observe that existing language mod- els rely heavily on word correlation and fall short of logical reasoning. This causes the model to hal- lucinate—for example, that Barack Obama’s wife is Hillary Clinton based on the high co-occurrence of the two entities. In another example, Lake and Murphy (2020) notice that GPT-2 (Radford et al., ∗Work completed during an internship at DeepMind. 2019) states that unicorns have four horns, directly after speaking that unicorns have one horn. In this work, we explore ways to combine an autoregressive language model with a knowledge graph. We design a memory-augmented archi- tecture that stores relations from a knowledge graph and investigate the effect of conditioning on this relational memory in an autoregressive lan- guage model. In contrast to existing token-based memory-augmented language models that store context-target pairs (Khandelwal et al., 2020b; Yogatama et al., 2021), our memory stores relation triples (head entity, relation, tail entity). Relation triples form the basis of knowledge bases, empow- ering a wide range of applications such as question answering (Yasunaga et al., 2021), machine read- ing (Yang and Mitchell, 2019), and reasoning (Minervini et al., 2020). From a cognitive science perspective, we can consider the neural language model to be an instance of System 1, which per- forms fast inference and the symbolic relational memory as a world model to support slow and log- ical reasoning of System 2 (Kahneman, 2011).1 We hypothesize that relational memory can im- prove performance and coherence of an autore- gressive language model. Given an observed context, we first run an en- tity tagger to identify …

Content truncated to 500 words