Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2259–2272 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2259 Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks Lisa Bauer UNC Chapel Hill lbauer6@cs.unc.edu Mohit Bansal UNC Chapel Hill mbansal@cs.unc.edu Abstract Integrating external knowledge into common- sense reasoning tasks has shown progress in resolving some, but not all, knowledge gaps in these tasks. For knowledge integration to yield peak performance, it is critical to select a knowledge graph (KG) that is well-aligned with the given task’s objective. We present an approach to assess how well a candidate KG can correctly identify and accurately fill in gaps of reasoning for a task, which we call KG-to-task match. to-task match in 3 phases:We show this KG- knowledge-task identification, knowledge-task alignment, and knowledge-task integration. We also analyze our transformer-based KG-to-task models via commonsense probes to measure how much knowledge is captured in these models be- fore and after KG integration. Empirically, we investigate KG matches for the SocialIQA (SIQA) (Sap et al., 2019b), Physical IQA (PIQA) (Bisk et al., 2020), and MCScript2.0 (Ostermann et al., 2019) datasets with 3 di- verse KGs: ATOMIC (Sap et al., 2019a), Con- ceptNet (Speer et al., 2017), and an automat- ically constructed instructional KG based on WikiHow (Koupaee and Wang, 2018). With our methods we are able to demonstrate that ATOMIC, an event-inference focused KG, is the best match for SIQA and MCScript2.0, and that the taxonomic ConceptNet and WikiHow- based KGs are the best matches for PIQA across all 3 analysis phases. We verify our methods and findings with human evaluation.1 1 Introduction Recently, several datasets (Sap et al., 2019b; Huang et al., 2019; Bhagavatula et al., 2020; Talmor et al., 2019b) have been released to tackle the challenge of commonsense reasoning. While deep pretrained 1Our code and commonsense probes will be publicly avail- able on our webpage. language-models (LMs) (Devlin et al., 2019; Rad- ford et al., 2019; Liu et al., 2019; Yang et al., 2019) have been at the top of most leaderboards, they still have shortcomings when it comes to commonsense reasoning (Sap et al., 2019b; Rajani et al., 2019; Mitra et al., 2019). Thus, incorporating knowledge graph (KG) information into these models is an active area of research (Lin et al., 2019a; Sun et al., 2019; Mitra et al., 2019; Bosselut et al., 2019). However, when selecting a KG match for a task, it is often difficult to quantitatively assess what kind of knowledge is missing from these models and how much of the missing knowledge required for the task is available in a candidate KG. It is also critical to examine how easily transformer-based models can learn commonsense knowledge, to de- termine the benefits of integrating a KG. We investigate how well a KG matches with a task objective, referred to as KG-to-task match. We use a 3-step process that examines knowledge identification, alignment, and integration. We uti- lize a …

Content truncated to 500 words