Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri?

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2583–2590 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2583 Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri? Olivier Ferret Universit´e Paris-Saclay, CEA, List, F-91120, Palaiseau, France olivier.ferret@cea.fr Abstract While contextual language models are now dominant in the field of Natural Language Processing, the representations they build at the token level are not always suitable for all uses. In this article, we propose a new method for building word or type-level embeddings from contextual models. This method combines the generalization and the aggregation of token representations. We evaluate it for a large set of English nouns from the perspective of the building of distributional thesauri for extracting semantic similarity relations. Moreover, we analyze the differences between static embeddings and type-level embeddings according to features such as the frequency of words or the type of semantic relations these embeddings account for, showing that the properties of these two types of embeddings can be complementary and exploited for further improving distributional thesauri. Keywords: static word embeddings, contextual word embeddings, semantic similarity, distributional thesauri 1. Introduction The introduction of contextual language models such as ELMo (Peters et al., 2018) or BERT (Devlin et al., 2019) in the field of Natural Language Processing rep- resents a major change in many dimensions. From the viewpoint of lexical semantics, one of them is the fact that these models produce representations at the token level instead of the word or type level. This change has generally a positive impact on classification or sequence labeling tasks that can be addressed by supervised ma- chine learning approaches. However, it raises more difficulties for tasks typically addressed by unsuper- vised approaches and focusing on the word level, such as the extraction of semantic relations between words for instance. One way to bypass these difficulties is to build type- level embeddings from a contextual language model1, which was already addressed by some studies. Etha- yarajh (2019) proposes to use principal component anal- ysis (PCA) as part of his analysis of the properties of contextual models while Bommasani et al. (2020) test a larger set of operations in the perspective of using type-level embeddings for investigating the properties of contextual models. The same kind of objective can also be found in (Vuli´c et al., 2020b) and (Vuli´c et al., 2020a) with a focus on semantic properties. Finally, Chronis and Erk (2020) explore the more specific issue of multi-prototype embeddings for accounting for the diversity of token representations. To some extent, the problem we consider is also linked to the building of meta-embeddings since the problem is to combine sev- 1Static embeddings such as those produced by the Skip- gram model (Mikolov et al., 2013) are intrinsically type-level embeddings but for avoiding confusion, we use in this paper the term static embeddings for referring to the embeddings produced by a non-contextual language model and the term type-level embeddings for referring …

Content truncated to 500 words