Learning Contextualized Knowledge Structures for Commonsense Reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4038–4051 August 1–6, 2021. ©2021 Association for Computational Linguistics 4038 Learning Contextualized Knowledge Structures for Commonsense Reasoning Jun Yan1, Mrigank Raman2, Aaron Chan1, Tianyu Zhang3, Ryan Rossi4, Handong Zhao4, Sungchul Kim4, Nedim Lipka4, Xiang Ren1 University of Southern California1, IIT Delhi2, Tsinghua University3, Adobe Research4 {yanjun, chanaaro, xiangren}@usc.edu, mt1170736@iitd.ac.in, zhang-ty17@mails.tsinghua.edu.cn, {ryrossi, hazhao, sukim, lipka}@adobe.com Abstract Recently, knowledge graph (KG) augmented models have achieved noteworthy success on various commonsense reasoning tasks. How- ever, KG edge (fact) sparsity and noisy edge extraction/generation often hinder mod- els from obtaining useful knowledge to rea- son over. To address these issues, we propose a new KG-augmented model: Hybrid Graph Network (HGN). Unlike prior methods, HGN learns to jointly contextualize extracted and generated knowledge by reasoning over both within a unified graph structure. Given the task input context and an extracted KG subgraph, HGN is trained to generate embeddings for the subgraph’s missing edges to form a “hybrid” graph, then reason over the hybrid graph while filtering out context-irrelevant edges. We demonstrate HGN’s effectiveness through con- siderable performance gains across four com- monsense reasoning benchmarks, plus a user study on edge validness and helpfulness.1 1 Introduction Commonsense reasoning (CSR) is essential for natural language understanding (NLU) systems to function effectively in the real world (Apperly, 2010). For example, to answer the question in Fig- ure 1, one must already know that printing requires using paper. Yet, since commonsense knowledge is self-evident to humans, it is rarely stated in natural language (Gunning, 2018). This makes it hard for neural pre-trained language models (PLMs) (De- vlin et al., 2019) to learn commonsense knowledge from corpora alone (Marcus, 2018). Unlike raw text corpora, knowledge graphs (KGs) can provide structured commonsense facts (edges) of the form (concept1, relation, concept2) (Speer et al., 2017). Hence, many re- cent CSR models augment the PLM with a KG, 1Our code and data can be found at https://github. com/INK-USC/HGN. Figure 1: KG-Augmented Commonsense QA. Pre- dicting the correct answer (“use paper”) requires com- monsense facts like (print, Requires, paper) and (paper, HasProperty, expensive), which are not given in the question and candidate an- swers. HGN uses facts extracted from the KG, e.g., (print, RelatedTo, use), but also generates new facts, eventually upweighting relevant ones, e.g., (print, Requires, use paper) and (paper, HasProperty, expensive), while downweighting irrelevant ones, e.g., (use, ?, expensive). allowing such KG-augmented models to make pre- dictions via multi-hop reasoning over the KG (Lin et al., 2019; Bosselut and Choi, 2019). Despite the growing success of KG-augmented models, obtaining helpful KG facts for a given task instance remains challenging. Existing mod- els assume using either KG-extracted edges (Lin et al., 2019; Ma et al., 2019; Feng et al., 2020; Ya- sunaga et al., 2021), PLM-generated edges (to ad- dress KG edge sparsity) (Bosselut and Choi, 2019), or a late fusion of both (Wang et al., 2020) is suf- ficient. Both extraction and generation can pro- duce unhelpful edges, so the model must …

Content truncated to 500 words