Probing for idiomaticity in vector space models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3551–3564 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3551 Probing for idiomaticity in vector space models Marcos Garcia CiTIUS Research Centre Universidade de Santiago de Compostela Galiza, Spain Tiago Kramer Vieira Federal University of Rio Grande do Sul, Brazil Carolina Scarton University of Sheffield, UK Marco Idiart Federal University of Rio Grande do Sul, Brazil marcos.garcia.gonzalez@udc.gal, tiagokv@hotmail.com, c.scarton@sheffield.ac.uk, marco.idiart@gmail.com, a.villavicencio@sheffield.ac.uk Aline Villavicencio University of Sheffield, UK Federal University of Rio Grande do Sul, Brazil Abstract Contextualised word representation models have been successfully used for capturing dif- ferent word usages, and they may be an at- tractive alternative for representing idiomatic- ity in language. In this paper, we propose probing measures to assess if some of the expected linguistic properties of noun com- pounds, especially those related to idiomatic meanings, and their dependence on context and sensitivity to lexical choice, are readily available in some standard and widely used representations. For that, we constructed the Noun Compound Senses Dataset, which con- tains noun compounds and their paraphrases, in context neutral and context informative nat- uralistic sentences, in two languages: English and Portuguese. Results obtained using four types of probing measures with models like ELMo, BERT and some of its variants, indi- cate that idiomaticity is not yet accurately rep- resented by contextualised models. 1 Introduction Contextualised word representation models, like BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018), seem to represent words more accurately than static word embeddings like GloVe (Penning- ton et al., 2014), as they can encode different usages of a word. In fact, representations of a word in sev- eral contexts can be grouped in different clusters, which seem to be related to the various senses of the word (Schuster et al., 2019), and they can be used to match polysemous words in context to specific sense definitions (Chang and Chen, 2019). How- ever, multiword expressions (MWEs) fall into a continuum of idiomaticity1 (Sag et al., 2002; Fazly 1We understand idiomaticity as semantic opacity and its continuum as different degrees of opacity (Cruse, 1986). et al., 2009; King and Cook, 2017) and their mean- ings may not be directly related to the meanings of their individual words (e.g., graduate student vs. eager beaver as a hardworking person). There- fore, one question is whether and to what extent idiomaticity in MWEs is accurately incorporated by word representation models. In this paper, we propose a set of probing mea- sures to examine how accurately idiomaticity in MWEs, particularly in noun compounds (NCs), is captured in vector space models, focusing on some widely used representations. Inspired by the se- mantic priming paradigm (Neely et al., 1989), we have designed four probing tasks to analyse how these models deal with some of the properties of NCs, including non-compositionality (big fish as an important person), non-substitutability (panda car vs. bear automobile), or ambiguity (bad apple as either a rotten fruit or …

Content truncated to 500 words