Dynamic Heterogeneous-Graph Reasoning with Language Models and Knowledge Representation Learning for Commonsense Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14048–14063 July 9-14, 2023 ©2023 Association for Computational Linguistics Dynamic Heterogeneous-Graph Reasoning with Language Models and Knowledge Representation Learning for Commonsense Question Answering Yujie Wang1, Hu Zhang1,2,∗, Jiye Liang1,2,∗, Ru Li1,2 1.School of Computer and Information Technology, Shanxi University, Taiyuan, China 2.Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, Taiyuan, China init_wang@foxmail.com,{zhanghu,ljy,liru}@sxu.edu.cn Abstract Recently, knowledge graphs (KGs) have won noteworthy success in commonsense question answering. Existing methods retrieve relevant subgraphs in the KGs through key entities and reason about the answer with language mod- els (LMs) and graph neural networks. How- ever, they ignore (i) optimizing the knowledge representation and structure of subgraphs and (ii) deeply fusing heterogeneous QA context with subgraphs. In this paper, we propose a dy- namic heterogeneous-graph reasoning method with LMs and knowledge representation learn- ing (DHLK), which constructs a heterogeneous knowledge graph (HKG) based on multiple knowledge sources and optimizes the structure and knowledge representation of the HKG us- ing a two-stage pruning strategy and knowl- edge representation learning (KRL). It then performs joint reasoning by LMs and Rela- tion Mask Self-Attention (RMSA). Specifically, DHLK filters key entities based on the dictio- nary vocabulary to achieve the first-stage prun- ing while incorporating the paraphrases in the dictionary into the subgraph to construct the HKG. Then, DHLK encodes and fuses the QA context and HKG using LM, and dynamically removes irrelevant KG entities based on the attention weights of LM for the second-stage pruning. Finally, DHLK introduces KRL to optimize the knowledge representation and per- form answer reasoning on the HKG by RMSA. We evaluate DHLK at CommonsenseQA and OpenBookQA, and show its improvement on existing LM and LM+KG methods. 1 Introduction Question answering (QA) is a challenging task that requires machines to understand questions asked by natural language and respond to the questions based on the knowledge acquired. Recently, QA has made remarkable progress with the develop- ment of Language Models (LMs) (Devlin et al., ∗Corresponding author. 2019; Liu et al., 2019; Lan et al., 2020; Raffel et al., 2020). Fine-tuning based on LMs has now become a major paradigm for QA tasks. LMs are pre-trained on a general large-scale corpus contain- ing rich world knowledge, which the machine can utilize when fine-tuning downstream tasks using LMs. In some simple, fact-based QA tasks, such as SQuAD (Rajpurkar et al., 2016, 2018) and RACE (Lai et al., 2017), machine has surpassed humans in terms of answer accuracy. However, the machine remains less satisfactory in some structured rea- soning QA tasks that require commonsense knowl- edge. Commonsense knowledge is the general law summarized by human beings through observation, research, and reflection of various phenomena in the objective world, which is verified by the long- term experience of countless people and is the com- mon daily consensus of people. When humans answer questions, they use this knowledge uncon- sciously. For example, if you ask …

Content truncated to 500 words