WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1635–1645 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1635 WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context Anna Breit1, Artem Revenko1, Kiamehr Rezaee2, Mohammad Taher Pilehvar3, Jose Camacho-Collados4 1Semantic Web Company, Austria; 2Iran University of Science and Technology; 3Tehran Institute for Advanced Studies, Iran; 4Cardiff University, United Kingdom 1{anna.breit,artem.revenko}@semantic-web.com, 2k rezaee@comp.iust.ac.ir, 3mp792@cam.ac.uk, 4camachocolladosj@cardiff.ac.uk Abstract We present WiC-TSV, a new multi-domain evaluation benchmark for Word Sense Disam- biguation. More specifically, we introduce a framework for Target Sense Verification of Words in Context which grounds its unique- ness in the formulation as binary classification task thus being independent of external sense inventories, and the coverage of various do- mains. This makes the dataset highly flexi- ble for the evaluation of a diverse set of mod- els and systems in and across domains. WiC- TSV provides three different evaluation set- tings, depending on the input signals provided to the model. We set baseline performance on the dataset using state-of-the-art language models. Experimental results show that even though these models can perform decently on the task, there remains a gap between machine and human performance, especially in out- of-domain settings. WiC-TSV data is avail- able at https://competitions.codalab. org/competitions/23683 1 Introduction Word Sense Disambiguation (WSD) is a long- standing task in Natural Language Processing and Artificial Intelligence. While progress has been made in recent years, mainly thanks to the surge of transformer-based language models such as BERT (Loureiro and Jorge, 2019; Vial et al., 2019; Huang et al., 2019), the evaluation of WSD models has been limited to a set of (mostly SemEval-based) standard WSD datasets (Raganato et al., 2017). These datasets usually come in one of the two forms: lexical sample, in which a target word is placed in various contexts, triggering different senses, and all-words, in which all the content words in a given text are to be disambiguated. Both settings, however, come with a major restriction: word senses in the datasets are linked to exter- nal sense inventories such as WordNet (Fellbaum, 1998). Therefore, existing benchmarks are limited to only those WSD systems in which sense dis- tinctions are defined according to an underlying sense inventory. This not only gives restrictions to the model’s flexibility, but also enforces the as- sumption of the availability of complete data. How- ever, as general sense inventories are complex to maintain they often lag behind in being up-to-date1, yielding to the absence of novel terms and term us- ages. Furthermore, the coverage of domain-specific terms and named entities in general sense invento- ries is quite limited, while domain-specific sense inventories are rare and in most cases incomplete. As a motivating example, let us assume Tech- nology as the target domain and the collection of information on the current technology landscape as a goal. Therefore, the following context needs to be disambiguated in order to evaluate its relevance: From 1970 to 2007, Apple’s chief …

Content truncated to 500 words