Modelling Commonsense Properties Using Pre-Trained Bi-Encoders

Proceedings of the 29th International Conference on Computational Linguistics, pages 3971–3983 October 12–17, 2022. 3971 Modelling Commonsense Properties using Pre-Trained Bi-Encoders Amit Gajbhiye∗, Luis Espinosa-Anke∗♢, Steven Schockaert∗ ∗CardiffNLP, Cardiff University, United Kingdom ♢AMPLYFI, United Kingdom {gajbhiyea, espinosa-ankel, schockaerts1}@cardiff.ac.uk Abstract Grasping the commonsense properties of ev- eryday concepts is an important prerequisite to language understanding. While contextu- alised language models are reportedly capa- ble of predicting such commonsense properties with human-level accuracy, we argue that such results have been inflated because of the high similarity between training and test concepts. This means that models which capture concept similarity can perform well, even if they do not capture any knowledge of the commonsense properties themselves. In settings where there is no overlap between the properties that are considered during training and testing, we find that the empirical performance of standard lan- guage models drops dramatically. To address this, we study the possibility of fine-tuning lan- guage models to explicitly model concepts and their properties. In particular, we train separate concept and property encoders on two types of readily available data: extracted hyponym- hypernym pairs and generic sentences. Our experimental1 results show that the resulting en- coders allow us to predict commonsense proper- ties with much higher accuracy than is possible by directly fine-tuning language models. We also present experimental results for the related task of unsupervised hypernym discovery. 1 Introduction Pre-trained language models (Devlin et al., 2019) have been found to capture a surprisingly rich amount of knowledge about the world (Petroni et al., 2019). Focusing on commonsense knowl- edge, Forbes et al. (2019) used BERT to predict whether a given concept (e.g. teddy bear) satisfies a given commonsense property (e.g. is dangerous). To this end, they convert the input into a simple sentence (e.g. “A teddy bear is dangerous”) and treat the task as a standard sentence classification 1Code and datasets are available at //github.com/amitgajbhiye/biencoder_https: concept_property task. Remarkably, they found this approach to sur- pass human performance. Shwartz and Choi (2020) moreover found that language models can, to some extent, capture commonsense properties that are rarely expressed in text, thus mitigating the issue of reporting bias that has traditionally plagued ini- tiatives for learning commonsense knowledge from text (Gordon and Durme, 2013). Despite these encouraging signs, however, mod- elling commonsense properties remains highly challenging. A key concern is that language mod- els are typically fine-tuned on a training set that contains the same properties as those in the test set. For instance, the test data from Forbes et al. (2019) includes the question whether peach has the property eaten in summer, while the training data asserts that apple, banana, orange and pear all have this property. To do well on this task, the model does not actually need to capture the knowledge that peaches are eaten in summer; it is sufficient to capture that peach is similar to apple, banana, orange and pear. For this reason, we pro- pose new training-test splits, which ensure that the properties occurring in the test …

Content truncated to 500 words