CoSimLex: A Resource for Evaluating Graded Word Similarity in Context

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5878–5886 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5878 CoSimLex: A Resource for Evaluating Graded Word Similarity in Context Carlos S. Armendariz∗, Matthew Purver∗†, Matej Ulˇcar‡, Senja Pollak†, Nikola Ljubeˇsi´c†, Marko Robnik-ˇSikonja‡, Mark Granroth-Wilding♦, Kristiina Vaik§ ∗Cognitive Science Research Group, Queen Mary University of London, London, UK {c.santosarmendariz, m.purver}@qmul.ac.uk †Department of Knowledge Technologies, Joˇzef Stefan Institute, Ljubljana, Slovenia {senja.pollak, nikola.ljubesic}@ijs.si ‡University of Ljubljana, Faculty of Computer and Information Science, Slovenia {matej.ulcar, marko.robnik}@fri.uni-lj.si ♦Department of Computer Science, University of Helsinki, Finland mark.granroth-wilding@helsinki.fi §Department of Data Analysis, Texta, Estonia kristiina.vaik@ut.ee Abstract State of the art natural language processing tools are built on context-dependent word embeddings, but no direct method for evaluating these representations currently exists. Standard tasks and datasets for intrinsic evaluation of embeddings are based on judgements of similarity, but ignore context; standard tasks for word sense disambiguation take account of context but do not provide continuous measures of meaning similarity. This paper describes an effort to build a new dataset, CoSimLex, intended to fill this gap. Building on the standard pairwise similarity task of SimLex-999, it provides context-dependent similarity measures; covers not only discrete differences in word sense but more subtle, graded changes in meaning; and covers not only a well-resourced language (English) but a number of less-resourced languages. We define the task and evaluation metrics, outline the dataset collection methodology, and describe the status of the dataset so far. Keywords: corpus, annotation, semantics, similarity, context, salience, context-dependence 1. Introduction Recent work in language modelling and word embeddings has led to a sharp increase in use of context-dependent mod- els such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). These models, by providing representations of words which depend on the surrounding context, allow us to take account of the effects not only of discrete dif- ferences in word sense but of the more graded effects of context. However, evaluation of these models has gener- ally been in terms of either their performance as language models, or their effect on downstream tasks such as senti- ment classification (Peters et al., 2018): there are few re- sources available which allow evaluation in terms of the properties of the embeddings themselves, or in terms of their ability to model human perceptions of meaning. There are established methods to evaluate word embedding mod- els intrinsically via their ability to reflect human similar- ity judgements (see e.g. WordSim-353 (Finkelstein et al., 2002) and SimLex-999 (Hill et al., 2015)) or model analo- gies (Mikolov et al., 2013); however, these have generally ignored context and treated words in isolation. The few that do provide context (e.g. SCWS (Huang et al., 2012) and WiC (Pilehvar and Camacho-Collados, 2019)) focus on word sense and discrete effects, thus missing some of the effects that context has on words in general, and some of the benefits of context-dependent models. To evaluate current models, we need a way to …

Content truncated to 500 words