Non-Autoregressive Translation by Learning Target Categorical Codes

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535–546 June 6–11, 2021. ©2021 Association for Computational Linguistics 535 QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering Michihiro Yasunaga Hongyu Ren Antoine Bosselut Percy Liang Jure Leskovec Stanford University {myasu,hyren,antoineb,pliang,jure}@cs.stanford.edu Abstract The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to esti- mate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions. 1 Introduction Question answering systems must be able to access relevant knowledge and reason over it. Typically, knowledge can be implicitly encoded in large language models (LMs) pre-trained on unstructured text (Petroni et al., 2019; Bosselut et al., 2019), or ex- plicitly represented in structured knowledge graphs (KGs), such as Freebase (Bollacker et al., 2008) and ConceptNet (Speer et al., 2017), where entities are represented as nodes and relations between them as edges. Recently, pre-trained LMs have demonstrated remarkable success in many question answering tasks (Liu et al., 2019; Raffel et al., 2020). However, while LMs have a broad coverage of knowledge, they do not empirically perform well on structured reasoning (e.g., handling negation) (Kassner and Schütze, 2020). On the other hand, KGs are more suited for structured reasoning (Ren et al., 2020; Ren and Leskovec, 2020) and enable explainable predictions e.g., by providing reasoning paths (Lin et al., 2019), but may lack coverage and If it is not used for hair, a round brush is an example of what? A. hair brush B. bathroom C. art supplies* D. shower E. hair salon painting AtLocation round brush hair brush RelatedTo AtLocation UsedFor UsedFor Choice Entity Answer Question Entity QA context Knowledge graph art supply QA context Node hair Graph Connection (§3.1) Dev Acc. No edge between Z and KG nodes 74.11 Connect Z to all KG nodes 76.38 Connect Z to QA entity nodes (final) 76.54 Relevance scoring (§3.2) Dev Acc. Nothing 75.15 w/ contextual embedding 76.31 w/ relevance score (final) 76.54 w/ both 76.52 GNN Attention & Message (§3.3) Dev Acc. Node type, relation, score-aware (final) 76.54 - type-aware 75.11 - relation-aware 75.23 - score-aware 75.15 GNN Layers (§3.3) Dev Acc. L = 3 75.53 L = 4 76.34 L = 5 (final) 76.54 L = 6 …

Content truncated to 500 words