Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9562 - 9581 December 7-11, 2022 ©2022 Association for Computational Linguistics Empowering Language Models with Knowledge Graph Reasoning for Question Answering Ziniu Hu1, Yichong Xu2, Wenhao Yu3, Shuohang Wang2 Ziyi Yang2, Chenguang Zhu2, Kai-Wei Chang1, Yizhou Sun1 1University of California, Los Angeles 2Microsoft Cognitive Services Research, 3University of Notre Dame Abstract Answering open-domain questions requires world knowledge about in-context entities. As pre-trained Language Models (LMs) lack the power to store all required knowledge, exter- nal knowledge sources, such as knowledge graphs, are often used to augment LMs. In this work, we propose knOwledge REasOning em- powered Language Model (OREOLM), which consists of a novel Knowledge Interaction Layer that can be flexibly plugged into existing Transformer-based LMs to interact with a differ- entiable Knowledge Graph Reasoning module collaboratively. In this way, LM guides KG to walk towards the desired answer, while the re- trieved knowledge improves LM. By adopting OREOLM to RoBERTa and T5, we show signif- icant performance gain, achieving state-of-art results in the Closed-Book setting. The per- formance enhancement is mainly from the KG reasoning’s capacity to infer missing relational facts. In addition, OREOLM provides reason- ing paths as rationales to interpret the model’s decision. 1 Introduction Open-Domain Question Answering (ODQA), one of the most knowledge-intensive NLP tasks, re- quires QA models to infer out-of-context knowl- edge to the given single question. Following the pioneering work by Chen et al. (2017), ODQA systems often assume to access an external text corpus (e.g., Wikipedia) as an external knowledge source. Due to the large scale of such textual knowl- edge sources (e.g., 20GB for Wikipedia), it cannot be encoded in the model parameters. Therefore, most works retrieve relevant passages as knowl- edge and thus named Open-Book models (Roberts et al., 2020), with an analogy of referring to text- books during an exam. Another line of Closed- book models (Roberts et al., 2020) assume knowl- edge could be stored implicitly in parameters of Language Models (LM, e.g. BERT (Devlin et al., Figure 1: An Illustrative figure of OREOLM. Compared with previous KBQA systems that stack reasoner on top of LM, OREOLM enables interaction between the two. 2019) and T5 (Raffel et al., 2020)). These LMs directly generate answers without retrieving from an external corpus and thus benefit from faster in- ference speed and simpler training. However, cur- rent LMs still miss a large portion of factual knowl- edge (Pörner et al., 2020; Lewis et al., 2021a), and are not competitive with Open-Book models. To improve the knowledge coverage of LM, one natural choice is to leverage knowledge stored in Knowledge Graph (KG, e.g. FreeBase (Bol- lacker et al., 2008) and WikiData (Vrandecic and Krötzsch, 2014)), which explicitly encodes world knowledge via relational triplets between entities. There are several good properties of KG: 1) a KG triplet is a more abstract and compressed represen- tation of knowledge than text, and thus KG could be stored in memory and directly enhance …

Content truncated to 500 words