Cambridge at SemEval-2021 Task 2: Neural WiC-Model with Data Augmentation and Exploration of Representation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 730–737 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 730 Cambridge at SemEval-2021 Task 2: Neural WiC-Model with Data Augmentation and Exploration of Representation Zheng Yuan and David Strohmaier ALTA Institute, University of Cambridge, United Kingdom Department of Computer Science and Technology, University of Cambridge, United Kingdom {zheng.yuan, david.strohmaier}@cl.cam.ac.uk Abstract This paper describes the system of the Cam- bridge team submitted to the SemEval-2021 shared task on Multilingual and Cross-lingual Word-in-Context Disambiguation. Building on top of a pre-trained masked language model, our system is first pre-trained on out-of- domain data, and then fine-tuned on in-domain data. We demonstrate the effectiveness of the proposed two-step training strategy and the benefits of data augmentation from both ex- isting examples and new resources. We fur- ther investigate different representations and show that the addition of distance-based fea- tures is helpful in the word-in-context disam- biguation task. Our system yields highly com- petitive results in the cross-lingual track with- out training on any cross-lingual data; and achieves state-of-the-art results in the multilin- gual track, ranking first in two languages (Ara- bic and Russian) and second in French out of 171 submitted systems. 1 Introduction Polysemy still poses a great challenge to natural language processing (NLP) applications. Depend- ing on its context, an ambiguous word can refer to multiple, potentially unrelated, meanings. Re- cently, as an application of Word Sense Disam- biguation (WSD) (Navigli, 2009, 2012), Word-in- Context (WiC) disambiguation has been framed as a binary classification task to identify if the occur- rences of a target word in two contexts correspond to the same meaning or not. The release of the WiC dataset (Pilehvar and Camacho-Collados, 2019), followed by the Multilingual Word-in-Context (XL- WiC) dataset (Raganato et al., 2020), has helped provide a common ground for evaluating and com- paring systems while encouraging research in WSD and context-sensitive word embeddings. In this paper, we describe our submission to the SemEval-2021 shared task on Multilingual and Cross-lingual Word-in-Context (MCL-WiC) Dis- ambiguation (Martelli et al., 2021), which involves determining whether a word shared by two sen- tences in the same language (multilingual track) or across different languages (cross-lingual track) has the same meaning in both contexts. Compared to previous WiC and XL-WiC benchmarks, two new languages are introduced as well as a cross- lingual track where systems are evaluated under a ‘zero-shot’ setting. The MCL-WiC task directly classifies pairs of sentences with regard to the meaning of the shared word. By turning WSD into a binary compari- son task, MCL-WiC avoids the need for sense tags of previous WSD shared tasks (Manandhar et al., 2010; Navigli et al., 2013; Moro and Navigli, 2015). It also resembles the Word Sense Alignment (WSA) task (Ahmadi et al., 2020) more closely, in which definitions from different dictionaries have to be aligned. Contextualised word embeddings and pre-trained Transformer-based (Vaswani et al., 2017) language models have been increasingly ap- plied to these tasks and state-of-the-art results have …

Content truncated to 500 words