Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 9–21 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 9 Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes Aina Gar´ı Soler1, Marianna Apidianaki1,2 and Alexandre Allauzen1 1LIMSI, CNRS, Univ. Paris Sud, Universit´e Paris-Saclay, F-91405 Orsay, France 2LLF, CNRS, Univ. Paris Diderot {aina.gari,marianna,allauzen}@limsi.fr Abstract Usage similarity estimation addresses the se- mantic proximity of word instances in differ- ent contexts. We apply contextualized (ELMo and BERT) word and sentence embeddings to this task, and propose supervised models that leverage these representations for predic- tion. Our models are further assisted by lexical substitute annotations automatically assigned to word instances by context2vec, a neural model that relies on a bidirectional LSTM. We perform an extensive comparison of existing word and sentence representations on bench- mark datasets addressing both graded and bi- nary similarity. The best performing models outperform previous methods in both settings. 1 Introduction Traditional word embeddings, like Word2Vec and GloVe, merge different meanings of a word in a single vector representation (Mikolov et al., 2013; Pennington et al., 2014). These pre-trained em- beddings are fixed, and stay the same indepen- dently of the context of use. Current contextual- ized sense representations, like ELMo and BERT, go to the other extreme and model meaning as word usage (Peters et al., 2018; Devlin et al., 2018). They provide a dynamic representation of word meaning adapted to every new context of use. In this work, we perform an extensive compar- ison of existing static and dynamic embedding- based meaning representation methods on the us- age similarity (Usim) task, which involves esti- mating the semantic proximity of word instances in different contexts (Erk et al., 2009). Usim differs from a classical Semantic Textual Simi- larity task (Agirre et al., 2016) by the focus on a particular word in the sentence. We evalu- ate on this task word and context representations obtained using pre-trained uncontextualized word Eleven CIRA members [have been [convicted of criminal charges and others are] awaiting trial]. So what started out as a perfectly lovely s as [a public wrestling match between g baby girl], with baby girl loving very s So what started out as a perfectly lovely stroll ended [up as [a public wrestling match between grandma and] baby girl], with baby girl loving very second of it . grass clippings can be brought out to the landfill at anytime for no *charge* and may not be placed in city cans . the tag consists of a tiny chip , [about the [size of a match head that serves ] as a ] portable database . this is at least 26 weeks by the [week in [which the approved match with the child ] is made ]. grass clippings can be brought out to the [landfill at [anytime for no charge and may not ] be placed in city cans ] Figure 1: We use contextualized word representations built from the whole sentence or smaller windows …

Content truncated to 500 words