Distilling Relation Embeddings from Pretrained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9044–9062 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9044 Distilling Relation Embeddings from Pre-trained Language Models Asahi Ushio and Jose Camacho-Collados and Steven Schockaert Cardiff NLP, School of Computer Science and Informatics Cardiff University, United Kingdom {UshioA,CamachoColladosJ,SchockaertS1}@cardiff.ac.uk Abstract Pre-trained language models have been found to capture a surprisingly rich amount of lexical knowledge, ranging from commonsense prop- erties of everyday concepts to detailed factual knowledge about named entities. Among oth- ers, this makes it possible to distill high-quality word vectors from pre-trained language mod- els. However, it is currently unclear to what ex- tent it is possible to distill relation embeddings, i.e. vectors that characterize the relationship between two words. Such relation embeddings are appealing because they can, in principle, encode relational knowledge in a more fine- grained way than is possible with knowledge graphs. To obtain relation embeddings from a pre-trained language model, we encode word pairs using a (manually or automatically gen- erated) prompt, and we fine-tune the language model such that relationally similar word pairs yield similar output vectors. We find that the resulting relation embeddings are highly competitive on analogy (unsupervised) and re- lation classification (supervised) benchmarks, even without any task-specific fine-tuning.1 1 Introduction One of the most widely studied aspects of word embeddings is the fact that word vector differences capture lexical relations (Mikolov et al., 2013a). While not being directly connected to downstream performance on NLP tasks, this ability of word em- beddings is nonetheless important. For instance, understanding lexical relations is an important pre- requisite for understanding the meaning of com- pound nouns (Turney, 2012). Moreover, the ability of word vectors to capture semantic relations has enabled a wide range of applications beyond NLP, including flexible querying of relational databases (Bordawekar and Shmueli, 2017), schema match- 1Source code to reproduce our experimental results and the model checkpoints are available in the following repository: https://github.com/asahi417/relbert ing (Fernandez et al., 2018), completion and re- trieval of Web tables (Zhang et al., 2019), ontology completion (Bouraoui and Schockaert, 2019) and information retrieval in the medical domain (Ar- guello Casteleiro et al., 2020). More generally, relational similarity (or analogy) plays a central role in computational creativity (Goel, 2019), le- gal reasoning (Ashley, 1988; Walton, 2010), on- tology alignment (Raad and Evermann, 2015) and instance-based learning (Miclet et al., 2008). Given the recent success of pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020), we may wonder whether such mod- els are able to capture lexical relations in a more faithful or fine-grained way than traditional word embeddings. However, for language models (LMs), there is no direct equivalent to the word vector difference. In this paper, we therefore propose a strategy for extracting relation embeddings from pre-trained LMs, i.e. vectors encoding the relation- ship between two words. On the one hand, this will allow us to gain a better understanding of how well lexical relations are captured by …

Content truncated to 500 words