WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

Proceedings of NAACL-HLT 2019, pages 1267–1273 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1267 WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations Mohammad Taher Pilehvar1,2 and Jose Camacho-Collados3 1DTAL, University of Cambridge, UK 2Tehran Institute for Advanced Studies (TeIAS), Tehran, Iran 3School of Computer Science and Informatics, Cardiff University, UK mp792@cam.ac.uk, camachocolladosj@cardiff.ac.uk Abstract By design, word embeddings are unable to model the dynamic nature of words’ seman- tics, i.e., the property of words to correspond to potentially different meanings. To address this limitation, dozens of specialized mean- ing representation techniques such as sense or contextualized embeddings have been pro- posed. However, despite the popularity of research on this topic, very few evaluation benchmarks exist that specifically focus on the dynamic semantics of words. In this paper we show that existing models have surpassed the performance ceiling of the standard evaluation dataset for the purpose, i.e., Stanford Contex- tual Word Similarity, and highlight its short- comings. To address the lack of a suitable benchmark, we put forward a large-scale Word in Context dataset, called WiC, based on anno- tations curated by experts, for generic evalua- tion of context-sensitive representations. WiC is released in https://pilehvar.github.io/wic/. 1 Introduction One of the main limitations of mainstream word embeddings lies in their static nature, i.e., a word is associated with the same embedding, indepen- dently from the context in which it appears. There- fore, these embeddings are unable to reflect the dynamic nature of ambiguous words1, in that they can correspond to different (potentially unrelated) meanings depending on their usage in context (Camacho-Collados and Pilehvar, 2018). To get around this limitation dozens of proposals have been put forward, mainly in two categories: multi- prototype embeddings (Reisinger and Mooney, 2010; Neelakantan et al., 2014; Pelevina et al., 2016), which usually leverage context clustering in order to learn distinct representations for in- dividual meanings of words, and contextualized 1Ambiguous words are important as they constitute the most frequent words in a natural language (Zipf, 1949). word embeddings (Melamud et al., 2016; Peters et al., 2018), which instead compute a single dy- namic embedding for a given word which can adapt itself to arbitrary contexts for the word. Despite the popularity of research on these spe- cialised embeddings, very few benchmarks ex- ist for their evaluation. Most works in this do- main either perform evaluations on word similar- ity datasets (in which words are presented in isola- tion; hence, they are not suitable for verifying the dynamic nature of word semantics) or carry out impact analysis in downstream NLP applications (usually, by taking word embeddings as baseline). Despite providing a suitable means of verifying the effectiveness of the embeddings, the down- stream evaluation cannot replace generic evalua- tions as it is difficult to isolate the impact of em- beddings from many other factors involved, in- cluding the algorithmic configuration and param- eter setting of the system. To our knowledge, the Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) is …

Content truncated to 500 words