Commonsense Knowledge Mining from Pretrained Models

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1173–1178, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1173 Commonsense Knowledge Mining from Pretrained Models Joshua Feldman∗, Joe Davison∗, Alexander M. Rush School of Engineering and Applied Sciences Harvard University {joshua feldman@g, jddavison@g, srush@seas}.harvard.edu Abstract Inferring commonsense knowledge is a key challenge in natural language processing, but due to the sparsity of training data, previ- ous work has shown that supervised methods for commonsense knowledge mining under- perform when evaluated on novel data. In this work, we develop a method for generat- ing commonsense knowledge using a large, pre-trained bidirectional language model. By transforming relational triples into masked sentences, we can use this model to rank a triple’s validity by the estimated pointwise mutual information between the two entities. Since we do not update the weights of the bidirectional model, our approach is not bi- ased by the coverage of any one common- sense knowledge base. Though this method performs worse on a test set than models ex- plicitly trained on a corresponding training set, it outperforms these methods when mining commonsense knowledge from new sources, suggesting that unsupervised techniques may generalize better than current supervised ap- proaches. 1 Introduction Commonsense knowledge consists of facts about the world which are assumed to be widely known. For this reason, commonsense knowledge is rarely stated explicitly in natural language, mak- ing it challenging to infer this information with- out an enormous amount of data (Gordon and Van Durme, 2013). Some have even argued that machine learning models cannot learn common sense implicitly (Davis and Marcus, 2015). One method for mollifying this issue is directly augmenting models with commonsense knowl- edge bases (Young et al., 2018), which typically contain high-quality information but with low cov- erage. These knowledge bases are represented as a graph, with nodes consisting of conceptual entities (i.e. dog, running away, excited, etc.) and the pre-defined edges representing the nature of the relations between concepts (IsA, UsedFor, CapableOf, etc.). Commonsense knowledge base completion (CKBC) is a machine learning task motivated by the need to improve the coverage of these resources. In this formulation of the prob- lem, one is supplied with a list of candidate entity- relation-entity triples, and the task is to distin- guish which of the triples express valid common- sense knowledge and which are fictitious (Li et al., 2016). Several approaches have been proposed for training models for commonsense knowledge base completion (Li et al., 2016; Jastrzebski et al., 2018). Each of these approaches uses some sort of supervised training on a particular knowl- edge base, evaluating the model’s performance on a held-out test set from the same database. These works use relations from ConceptNet, a crowd-sourced database of structured common- sense knowledge, to train and validate their mod- els (Liu and Singh, 2004). However, it has been shown that these methods generalize poorly to novel data (Li et al., …

Content truncated to 500 words