Adaptable and Interpretable Neural MemoryOver Symbolic Knowledge

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3678–3691 June 6–11, 2021. ©2021 Association for Computational Linguistics 3678 Adaptable and Interpretable Neural Memory Over Symbolic Knowledge Pat Verga*, Haitian Sun*, Livio Baldini Soares, William W. Cohen Google Research {patverga, haitiansun, liviobs, wcohen}@google.com Abstract Past research has demonstrated that large neu- ral language models (LMs) encode surpris- ing amounts of factual information: how- ever, augmenting or modifying this informa- tion requires modifying a corpus and retrain- ing, which is computationally expensive. To address this problem, we develop a neural LM that includes an interpretable neuro-symbolic KB in the form of a “fact memory”. Each element of the fact memory is formed from a triple of vectors, where each vector corre- sponds to a KB entity or relation. Our LM im- proves performance on knowledge-intensive question-answering tasks, sometimes dramat- ically, including a 27 point increase in one set- ting of WebQuestionsSP over a state-of-the-art open-book model, despite using 5% of the pa- rameters. Most interestingly, we demonstrate that the model can be modified, without any re-training, by updating the fact memory. 1 Introduction Neural language models (LMs) (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2019) that have been pre-trained by self-supervision on large cor- pora contain rich knowledge about the syntax and semantics of natural language (Tenney et al., 2019), and are the basis of much recent work in NLP. Pre- trained LMs also contain large amounts of factual knowledge about the world (Petroni et al., 2019; Roberts et al., 2020; Brown et al., 2020). However, while large LMs can be coerced to answer factual queries, they still lack many of the properties that knowledge bases (KBs) typically have. In particu- lar, it is difficult to distinguish answers produced by memorizing factual statements in the pre-training corpus from lower-precision answers produced by linguistic generalization (Poerner et al., 2019). It is also difficult to add or remove factual informa- tion without retraining the LM, an expensive pro- cess1. The difficulty of updating knowledge in neural LMs contrasts with symbolic KBs, where it is very easy to add or modify triples, and is a major disadvantage of using a LM “as a KB”—as in many domains (news, product reviews, scientific publications, etc) the set of known facts changes frequently. Symbolic KBs thus remain practically important (Google, 2012; Dong, 2017), especially for NLP applications where text is hard to automat- ically process (e.g., scientific, technical, or legal) or tasks rich in information that exists only in struc- tured form (e.g., technical specifications of a new product, where no product page or review text dis- cussing it yet exists). Motivated by this, past work has sought to com- bine the benefits of neural LMs with the large, broad-coverage KBs that now exist (Bollacker et al., 2008; Auer et al., 2007; Vrandeˇci´c and Krötzsch, 2014). This paper continues this re- search program with a new knowledge-augmented LM called Fact Injected Language …

Content truncated to 500 words