Do Language Models Perform Generalizable Commonsense Inference?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3681–3688 August 1–6, 2021. ©2021 Association for Computational Linguistics 3681 Do Language Models Perform Generalizable Commonsense Inference? Peifeng Wang1,2, Filip Ilievski2, Muhao Chen1,2, Xiang Ren1,2 1Department of Computer Science, University of Southern California 2Information Sciences Institute, University of Southern California {peifengw,muhaoche,xiangren}@usc.edu, ilievski@isi.edu Abstract Inspired by evidence that pretrained language models (LMs) encode commonsense knowl- edge, recent work has applied LMs to auto- matically populate commonsense knowledge graphs (CKGs). However, there is a lack of understanding on their generalization to mul- tiple CKGs, unseen relations, and novel enti- ties. This paper analyzes the ability of LMs to perform generalizable commonsense infer- ence, in terms of knowledge capacity, transfer- ability, and induction. Our experiments with these three aspects show that: (1) LMs can adapt to different schemas defined by multiple CKGs but fail to reuse the knowledge to gen- eralize to new relations. (2) Adapted LMs gen- eralize well to unseen subjects, but less so on novel objects. Future work should investigate how to improve the transferability and induc- tion of commonsense mining from LMs.1 1 Introduction Large-scale commonsense knowledge graphs (CKGs), like ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019), store structured knowl- edge that can benefit various knowledge-driven applications. Given the usefulness of CKGs, but also their inability to flexibly provide information, (Paulheim, 2018), recent work has paid much at- tention to populating CKGs with commonsense knowledge mined from pretrained language models (LMs) (Wang et al., 2020c; Bosselut et al., 2019). Enhancing the knowledge of CKGs is essential to support reasoning on downstream tasks (Talmor et al., 2019; Wang et al., 2020b; Young et al., 2018). The task of completing CKGs has typically been posed as commonsense knowledge inference, where the goal is to predict the object of a fact triplet, given its subject and a relation (predicate) (Petroni 1The code is avaiable at https://github.com/ wangpf3/LM-for-CommonsenseInference. 0. Single CKG 1. Multi-task 2. Transfer Learning 3. Low-resource Figure 1: Unlike previous studies that adapt LM on one single CKG (0), we investigate LM’s three aspects of generlizability: (1) knowledge capacity by multi-task learning, (2) transferability by transfer learning and (3) induction by controlled low-resource learning. et al., 2019; Bosselut et al., 2019). Commonsense inference techniques, such as COMET (Bosse- lut et al., 2019), typically fine-tune an LM, like GPT (Radford et al., 2018), over the training set from a single CKG. While such methods are able to dynamically enhance the completeness of CKGs, their application so far has been limited to the re- lation set of the source (training) CKG (Da et al., 2021). In addition, the generated object concepts are found to be largely biased towards the ones in the training set (Wang et al., 2020a). It remains unclear to which extent LMs can generalize to mul- tiple CKGs, new relations, and novel objects. To this end, we pose the question: do language models perform generalizable commonsense inference? To answer this question, we study three aspects of …

Content truncated to 500 words