Sentence Selection Strategies for Distilling Word Embeddings from BERT

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2591–2600 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2591 Sentence Selection Strategies for Distilling Word Embeddings from BERT Yixiao Wang1, Zied Bouraoui2, Luis Espinosa-Anke1, Steven Schockaert1 1 Cardiff University, UK, {wangy306, espinosa-ankel, schockaerts1}@cardiff.ac.uk 2CRIL-CNRS, Universit´e d’Artois, France, zied.bouraoui@cril.fr Abstract Many applications crucially rely on the availability of high-quality word vectors. To learn such representations, several strategies based on language models have been proposed in recent years. While effective, these methods typically rely on a large number of contextualised vectors for each word, which makes them impractical. In this paper, we investigate whether similar results can be obtained when only a few contextualised representations of each word can be used. To this end, we analyze a range of strategies for selecting the most informative sentences. Our results show that with a careful selection strategy, high-quality word vectors can be learned from as few as 5 to 10 sentences. Keywords: Word Embeddings, Language Models, Natural Language Processing 1. Introduction The ability to model word meaning in context is a central feature of transformer-based language models. Nonetheless, since the introduction of BERT (Devlin et al., 2019), various authors have also explored the possibility of using language models for learning bet- ter static word embeddings (Ethayarajh, 2019; Bom- masani et al., 2020; Vuli´c et al., 2020; Gan et al., 2020; Li et al., 2021; Gupta and Jaggi, 2021). Distilling static word vectors from language models is useful for sev- eral reasons. Some authors have focused on analyzing these vectors to better understand how BERT captures word meaning (Ethayarajh, 2019), and for investigat- ing the social biases of such models (Bommasani et al., 2020). Liu et al. (2020) found that static word vectors can be useful as “anchors” for improving the representations of language models, while Alghanmi et al. (2020) also obtained improved results by combin- ing BERT with static word vectors. Another motiva- tion comes from application settings where efficiency and/or transparency is key (Gupta and Jaggi, 2021). However, our main motivation comes from applica- tions where word meaning has to be modelled in the absence of any sentence context. Examples of such ap- plications include ontology alignment (Kolyvakis et al., 2018), ontology completion (Li et al., 2019), and zero- shot (Socher et al., 2013; Ma et al., 2016) and few-shot (Xing et al., 2019; Li et al., 2020; Yan et al., 2021) learning. The most common strategy for distilling word vectors from BERT is based on sampling sentences that men- tion the words of interest. To obtain an embedding of w, we can then simply average the contextualised representations of w across these sentences. Specific strategies differ in (i) which layers of the transformer model are being used and (ii) whether or not the word w is masked. The issue of masking w was studied in particular by Li et al. (2021), who found this to lead to representations that were better suited to …

Content truncated to 500 words