Using Distributional Principles for the Semantic Study of Contextual Language Models

Using Distributional Principles for the Semantic Study of Contextual Language Models Olivier Ferret Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France olivier.ferret@cea.fr Abstract Many studies were recently done for investi- gating the properties of contextual language models but surprisingly, only a few of them consider the properties of these models in terms of semantic similarity. In this article, we first focus on these properties for English by ex- ploiting the distributional principle of substitu- tion as a probing mechanism in the controlled context of SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same method to a more open setting for characteriz- ing the differences between static and contex- tual language models. 1 Introduction The introduction of contextual word embeddings such as ELMo (Peters et al., 2018) or BERT (Devlin et al., 2019) is considered as a major breakthrough in Natural Language Processing, especially for clas- sification or sequence labeling tasks that can be ad- dressed by supervised machine learning approaches. However, this kind of embeddings also represents a significant change from the viewpoint of distribu- tional semantics. While previous approaches, includ- ing static word embeddings such as Skip-gram or CBOW models (Mikolov et al., 2013), built the dis- tributional representation of a word by cumulating the contexts in which it was found, the neural lan- guage models such as ELMo or BERT produce a representation for each occurrence of the word. While this specificity is not a difficulty – and even more so an advantage – in the context of a super- vised classification or sequence labeling task, it is a problem in contexts requiring a distributional repre- sentation of words at the type level and not only at the token level. This is the case for the evaluation of the semantic similarity of words, which is a classi- cal tool for investigating the semantic properties of distributional representations through datasets, such as Simlex-99 (Hill et al., 2015), built for evaluating the correlation of the similarity computed from these representations with human judgments. As illustrated by Rogers et al. (2020), the proper- ties of contextual word embeddings have been the focus of many recent studies, especially in the case of BERT. Surprisingly, only a few of them consider the semantic properties of these models. Moreover, most of them are performed at the word type level, which involves building representations of words from the representation of their occurrences, which is neither a straightforward nor a neutral operation. In this article, we propose to exploit the distribu- tional principles to investigate the semantic properties of contextual word embeddings in terms of paradig- matic semantic relations without the requirement to build representations at the word type level. More precisely, we present the following main contribu- tions: • we show that the distributional hypothesis can be applied at the word token level with contextual word embeddings and that ELMo and BERT exhibit specific semantic properties falling more into the scope of semantic similar- ity than semantic relatedness; • we show how the …

Content truncated to 500 words