Structured Self-Supervised Pretraining for Commonsense Knowledge Graph Completion

Structured Self-Supervised Pretraining for Commonsense Knowledge Graph Completion Jiayuan Huang•∗, Yangkai Du•∗, Shuting Tao•∗, Kun Xu⋄, Pengtao Xie⋆† •Zhejiang University, China, ⋄Tencent AI Lab, USA, ⋆UC San Diego, USA p1xie@eng.ucsd.edu Abstract To develop commonsense-grounded NLP ap- plications, a comprehensive and accurate com- monsense knowledge graph (CKG) is needed. It is time-consuming to manually construct CKGs and many research efforts have been de- voted to the automatic construction of CKGs. Previous approaches focus on generating con- cepts that have direct and obvious relationships with existing concepts and lack an capability to generate unobvious concepts. In this work, we aim to bridge this gap. We propose a gen- eral graph-to-paths pretraining framework that leverages high-order structures in CKGs to capture high-order relationships between con- cepts. We instantiate this general framework to four special cases: long path, path-to-path, router, and graph-node-path. Experiments on two datasets demonstrate the effectiveness of our methods. The code will be released via the public GitHub repository. 1 Introduction Commonsense knowledge has been widely used to boost many NLP applications, such as dialog generation (Zhou et al., 2018), question answering (Talmor et al., 2018), story generation (Guan et al., 2020), and so forth. To ground an application with commonsense, one needs to access a common- sense knowledge graph (CKG) where nodes repre- sent concepts such as ‘‘maintain muscle strength’’, ‘‘exercise regularly’’, and edges represent the re- lationships between concepts such as ‘‘maintain muscle strength’’ has a prerequisite of ‘‘exercise regularly’’. Commonsense knowledge involves almost all concepts in a human’s daily life. These concepts have very rich and diverse relationships. As a result, it is extremely challenging, if not impossi- ble, for humans to list all commonsense concepts and relationships exhaustively. To address this is- ∗Equal contribution. †Corresponding author. sue, many efforts (Malaviya et al., 2020; Bosselut et al., 2019) have been devoted to automatically constructing CKGs. A commonly used approach is: Given a head concept and a relation, train a generative model to generate the tail concept. Typically, the generative model consists of an encoder that encodes the concatenation of head concept and relation, and a decoder that takes the embedding generated by the encoder as input and decodes a tail concept. While simple, these ap- proaches treat a CKG as a collection of individual concept-relation-concept triples without consid- ering the rich structures in a CKG. As a result, although existing methods can generate new con- cepts that have direct and obvious relationships with existing concepts in a CKG, they are lack- ing in generating concepts that are indirectly and unobviously related to existing concepts. This problem can be potentially addressed by exploiting the rich structures in a CKG to perform high-order reasoning that helps to generate un- obvious concepts. Figure 1 shows an example. Previous approaches are able to generate obvious concepts/relations such as generating ‘‘hiking’’ given ‘‘fatigue’’ and ‘‘is caused by’’ and generat- ing ‘‘sleep’’ given ‘‘fatigue’’ and ‘‘makes people want to’’. However, since the relationship between ‘‘hiking’’ and ‘‘sleep’’ is not obvious, previous approaches are unlikely …

Content truncated to 500 words