WiC-TSV-de: German Word-in-Context Target-Sense-Verification Dataset and Cross-Lingual Transfer Analysis

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2617–2625 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2617 WiC-TSV-de: German Word-in-Context Target-Sense-Verification Dataset and Cross-Lingual Transfer Analysis Anna Breit, Artem Revenko, Narayani Blaschke Semantic Web Company Neubaugasse 1, Vienna, Austria {anna.breit, artem.revenko}@semantic-web.com Abstract Target Sense Verification (TSV) describes the binary disambiguation task of deciding whether the intended sense of a target word in a context corresponds to a given target sense. In this paper, we introduce WiC-TSV-de, a multi-domain dataset for German Target Sense Verification. While the training and development sets consist of domain-independent instances only, the test set contains domain-bound subsets, originating from four different domains, being Gastronomy, Medicine, Hunting, and Zoology. The domain-bound subsets incorporate adversarial examples such as in-domain ambiguous target senses and context-mixing (i.e., using the target sense in an out-of-domain context) which contribute to the challenging nature of the presented dataset. WiC-TSV-de allows for the development of sense-inventory-independent disambiguation models that can generalise their knowledge for different domain settings. By combining it with the original English WiC-TSV benchmark, we performed monolingual and cross-lingual analysis, where the evaluated baseline models were not able to solve the dataset to a satisfying degree, leaving a big gap to human performance. WiC-TSV-de data is openly available at https://github.com/semantic-web-company/wic-tsv/. Keywords: Target Sense Verification, Disambiguation, German, Dataset, Cross-Lingual Disambiguation 1. Introduction Being able to distinguish between different meanings of a word is a crucial pre-processing step for a wide va- riety of down-stream tasks such as document retrieval, sentiment analysis, or relation extraction. Traditionally, the formulation of the Word Sense Dis- ambiguation (WSD) task was used to tackle this prob- lem, where for a given target word, the corresponding best-matching entry from an external word sense in- ventory was retrieved. Though this task formulation enjoys great popularity, it comes with downsides that especially show in domain-specific, real-world scenar- ios: since the target senses are linked to external inven- tories, WSD systems need to be modelled accordingly to these resources, which not only reduces the flexibil- ity of the models, but also enforces the assumption of the availability of complete data within these resources. In certain scenarios however, it might be impractical or even impossible to model all possible senses. The more recent task formulation of Word-in-Context (WiC) breaks with this dependencies on external sense inventories by focusing on the question whether two contexts use the target word in the same sense. This binary formulation increases the flexibility, but focuses on identifying similar usages of a target instead of as- signing a specific sense to a target. In domain-specific and enterprise settings, the focus often lies on a defined set of concepts, i.e., a relatively small subset of all pos- sible senses, which needs to be identified and disam- biguated within textual data. The task formulation of Target Sense Verification (TSV) combines the independence of sense inventories and the possibility to identify and assign specific senses, by formulating the disambiguation as a …

Content truncated to 500 words