GlossReader at SemEval-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 756–762 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 756 GlossReader at SemEval-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings Maxim Rachinskiy♦ and Nikolay Arefyev△∇♦ ♦HSE University, Moscow, Russia △Samsung Research Center Russia, Moscow, Russia ∇Lomonosov Moscow State University, Moscow, Russia myurachinskiy@edu.hse.ru narefjev@cs.msu.ru Abstract Consulting a dictionary or a glossary is a fa- miliar way for many humans to figure out what does a word in a particular context mean. We hypothesize that a system that can se- lect a proper definition for a particular word occurrence can also naturally solve tasks re- lated to word senses. To verify this hypoth- esis we developed a solution for the Mul- tilingual and Cross-lingual Word-in-Context (MCL-WiC) task, that does not use any of the shared task data or other WiC data for training. Instead, it is trained to embed word definitions from English WordNet and word occurrences in English texts into the same vector space following an approach previously proposed by Blevins and Zettlemoyer (2020) for Word Sense Disambiguation (WSD). To estimate the similarity in meaning of two word occurrences, we compared different metrics in this shared vector space and found that L1-distance be- tween normalized contextualized word embed- dings outperforms traditionally employed co- sine similarity and several other metrics. To solve the task for languages other than English, we rely on zero-shot cross-lingual transfer ca- pabilities of the multilingual XLM-R masked language model. Despite not using MCL-WiC training data, in the shared task our approach achieves an accuracy of 89.5% on the En- glish test set, which is only 4% less than the best system. In the multilingual subtask zero- shot cross-lingual transfer shows competitive results, that are within 2% from the best sys- tems for Russian, French, and Arabic. In the cross-lingual subtask are within 2-4% from the best systems. 1 Introduction SemEval-2021 Task 2 is a multilingual and cross- lingual word-in-context disambiguation task (MCL- WiC) for five different languages (Martelli et al., 2021). 1 Each example in the multilingual sub- task consists of two sentences in English, Rus- sian, French, Arabic, or Chinese language con- taining occurrences of the same target word. In the cross-lingual subtask each example consists of two sentences in different languages contain- ing occurrences of two different target words. The participants were asked to detect whether those oc- currences corresponded to the same or different meanings. These tasks are formalized as binary classification tasks. The datasets contain the same number of examples for each class. Accuracy is utilized as the main performance metric. Recent SOTA approaches to the WiC task mainly include fine-tuning large universally pre-trained masked language models (Raffel et al., 2020; Liu et al., 2019) on labeled WiC datasets. Instead, we decided to train a system that selects the most appropriate definition for each word occurrence following an approach proposed by Blevins and Zettlemoyer (2020) for Word Sense Disambigua- tion (WSD) and experimented with different ways of adapting …

Content truncated to 500 words