CoSe-Co: Text Conditioned Generative CommonSense Contextualizer

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1128 - 1143 July 10-15, 2022 ©2022 Association for Computational Linguistics CoSe-Co: Text Conditioned Generative CommonSense Contextualizer Rachit Bansal1∗ Milan Aggarwal2 Sumit Bhatia2 Jivat Neet Kaur3∗ Balaji Krishnamurthy2 1 Delhi Technological University 2 Media and Data Science Research Lab, Adobe, India 3 Microsoft Research, India racbansa@gmail.com, {milaggar, sumbhati, kbalaji}@adobe.com jivatneet@gmail.com Abstract Pre-trained Language Models (PTLMs) have been shown to perform well on natural lan- guage tasks. Many prior works have lever- aged structured commonsense present in the form of entities linked through labeled rela- tions in Knowledge Graphs (KGs) to assist PTLMs. Retrieval approaches use KG as a separate static module which limits coverage since KGs contain finite knowledge. Gener- ative methods train PTLMs on KG triples to improve the scale at which knowledge can be obtained. However, training on symbolic KG entities limits their applicability in tasks in- volving natural language text where they ig- nore overall context. To mitigate this, we pro- pose a CommonSense Contextualizer (CoSe- Co) conditioned on sentences as input to make it generically usable in tasks for generating knowledge relevant to the overall context of input text. To train CoSe-Co, we propose a novel dataset comprising of sentence and com- monsense knowledge pairs. The knowledge inferred by CoSe-Co is diverse and contain novel entities not present in the underlying KG. We augment generated knowledge in Multi- Choice QA and Open-ended CommonSense Reasoning tasks leading to improvements over current best methods on CSQA, ARC, QASC and OBQA datasets. We also demonstrate its applicability in improving performance of a baseline model for paraphrase generation task. 1 Introduction While dealing with natural language text, common- sense allows humans to expand salient concepts and infer additional information. For example, by reading a sign like Men at Work on a road, we implicitly know to slow down our vehicles, look carefully for workers. This implicit process of us- ing common sense to make logical inferences is critical to natural language understanding (Xie and ∗Work done as an intern at the Media and Data Science Research Lab, Adobe, India Pu, 2021). A natural question to ask then is how we can incorporate common sense in now-ubiquitous language models (LMs) (Devlin et al., 2019; Rad- ford et al., 2018a; Raffel et al., 2019). There have been various efforts (Bao et al., 2016; Feng et al., 2020; Wang et al., 2020b) to lever- age structured knowledge present in commonsense knowledge graphs - KGs (we use KG as a short- hand for Commonsense Knowledge Graph) (Xie and Pu, 2021). Such works have primarily focused on either retrieving or generating required knowl- edge. Retrieval methods rely heavily on structure of downstream task like multi-choice question answer- ing (QA) to leverage knowledge in a KG (Yasunaga et al., 2021) and are not applicable beyond a spe- cific task. Further, retrieval can restrict total knowl- edge that can be garnered since static KGs lack coverage due …

Content truncated to 500 words