JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5049 - 5060 July 10-15, 2022 ©2022 Association for Computational Linguistics JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering Yueqing Sun, Qi Shi, Le Qi, Yu Zhang∗ Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, Harbin, China {yqsun,qshi,lqi,zhangyu}@ir.hit.edu.cn Abstract Existing KG-augmented models for common- sense question answering primarily focus on designing elaborate Graph Neural Networks (GNNs) to model knowledge graphs (KGs). However, they ignore (i) the effectively fus- ing and reasoning over question context repre- sentations and the KG representations, and (ii) automatically selecting relevant nodes from the noisy KGs during reasoning. In this paper, we propose a novel model, JointLK, which solves the above limitations through the joint reason- ing of LM and GNN and the dynamic KGs pruning mechanism. Specifically, JointLK per- forms joint reasoning between LM and GNN through a novel dense bidirectional attention module, in which each question token attends on KG nodes and each KG node attends on question tokens, and the two modal represen- tations fuse and update mutually by multi-step interactions. Then, the dynamic pruning mod- ule uses the attention weights generated by joint reasoning to prune irrelevant KG nodes recursively. We evaluate JointLK on the Com- monsenseQA and OpenBookQA datasets, and demonstrate its improvements to the existing LM and LM+KG models, as well as its capa- bility to perform interpretable reasoning1. 1 Introduction Commonsense question answering (CSQA) re- quires systems to acquire different types of com- monsense knowledge and reasoning skills, which is normal for humans, but challenging for machines (Talmor et al., 2019). Recently, large pre-trained language models (LMs) have achieved remarkable success in many QA tasks and appear to use im- plicit (factual) knowledge encoded in their model parameters during fine-tuning (Liu et al., 2019; Raffel et al., 2020). Nevertheless, commonsense ∗Corresponding author. 1Our code is available at: https://github.com/Yueqing- Sun/JointLK Text Encoder Graph Encoder Score wood play_guitar guitar … take_lessons action singer burn dance gas singing fun Question: What do people typically do while playing guitar? Choices: A. cry B. hear sounds C. singing* D. arthritis E. making music Joint Reasoning Subgraph singe Figure 1: Our knowledge-augmented joint reasoning model framework with an example from Common- senseQA. The subgraph is retrieved from ConceptNet. knowledge is self-evident to humans and is rarely expressed clearly in natural language (Gunning, 2018), which makes it difficult for LMs to learn commonsense knowledge from the pre-training text corpus alone. An extensive research path is to elaborately de- sign graph neural networks (GNNs) (Scarselli et al., 2008) to perform reasoning over explicit structural common sense knowledge from external knowl- edge bases (Vrandeˇci´c and Krötzsch, 2014; Speer et al., 2017). Related methods usually follow a retrieval-and-modeling paradigm. First, the knowl- edge subgraphs or paths related to a given question are retrieved by string matching or semantic similar- ity; such retrieved structured information indicates the relation between concepts or …

Content truncated to 500 words