XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7193–7206, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7193 XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization Alessandro Raganato⋆♥ Tommaso Pasini⋆♦ Jose Camacho-Collados♣ Mohammad Taher Pilehvar♠ ♥ Department of Digital Humanities, University of Helsinki, Finland ♦ SapienzaNLP Group, Computer Science Department, Sapienza University of Rome, Italy ♣ School of Computer Science and Informatics, Cardiff University, United Kingdom ♠ Tehran Institute for Advanced Studies, Iran ♥alessandro.raganato@helsinki.fi, ♦pasini@di.uniroma1.it, ♣camachocolladosj@cardiff.ac.uk, ♠mp792@cam.ac.uk Abstract The ability to correctly model distinct mean- ings of a word is crucial for the effectiveness of semantic representation techniques. How- ever, most existing evaluation benchmarks for assessing this criterion are tied to sense inven- tories (usually WordNet), restricting their us- age to a small subset of knowledge-based rep- resentation techniques. The Word-in-Context dataset (WiC) addresses the dependence on sense inventories by reformulating the stan- dard disambiguation task as a binary classifi- cation problem; but, it is limited to the En- glish language. We put forward a large mul- tilingual benchmark, XL-WiC, featuring gold standards in 12 new languages from varied lan- guage families and with different degrees of resource availability, opening room for evalu- ation scenarios such as zero-shot cross-lingual transfer. We perform a series of experiments to determine the reliability of the datasets and to set performance baselines for several recent contextualized multilingual models. Experi- mental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance in the task of distinguishing different meanings of a word, even for distant languages. XL-WiC is avail- able at https://pilehvar.github.io/xlwic/. 1 Introduction One of the desirable properties of contextualized models, such as BERT (Devlin et al., 2019) and its derivatives, lies in their ability to associate dynamic representations to words, i.e., embeddings that can change depending on the context. This provides the basis for the model to distinguish different mean- ings (senses) of words without the need to resort to an explicit sense disambiguation step. The conven- tional evaluation framework for this property has Authors marked with a star (⋆) contributed equally. been Word Sense Disambiguation (Navigli, 2009, WSD). However, evaluation benchmarks for WSD are usually tied to external sense inventories (often WordNet (Fellbaum, 1998)), making it extremely difficult to evaluate systems that do not explicitly model sense distinctions in the inventory, effec- tively restricting the benchmark to inventory-based sense representation techniques and WSD systems. This prevents a direct evaluation of lexical semantic capacity for a wide range of inventory-free mod- els, such as the dominating language model-based contextualized representations. Pilehvar and Camacho-Collados (2019) ad- dressed this dependence on sense inventories by reformulating the WSD task as a simple binary classification problem: given a target word w in two different contexts, c1 and c2, the task is to identify if the same meaning (sense) of w was in- tended in both c1 and c2, or not. The task was framed as a dataset, called …

Content truncated to 500 words