PAW at SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation : Exploring Cross Lingual Transfer, Augmentations and Adversarial Training

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 743–747 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 743 PAW at SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation : Exploring Cross Lingual Transfer, Augmentations and Adversarial Training Harsh Goyal, Aadarsh Singh and Priyanshu Kumar Indian Institute of Technology (Indian School of Mines) Dhanbad, India {harshgoyal13, aadarshsingh191198, kpriyanshu256}@gmail.com Abstract We experiment with XLM RoBERTa for Word in Context Disambiguation in the Multi Lin- gual and Cross Lingual setting so as to de- velop a single model having knowledge about both settings. We solve the problem as a bi- nary classification problem and also experi- ment with data augmentation and adversarial training techniques. In addition, we also exper- iment with a 2-stage training technique. Our approaches prove to be beneficial for better performance and robustness. 1 Introduction Language is complex even for human beings, let alone for computers. The same word serves differ- ent purposes in different scenarios, thus increasing the complexity of the Word Sense Disambiguation (WSD). For example in English, the word “bank” can refer to a financial institution or the land along- side a river. Many works revolving around WSD have been done with the help of explicit word sense inventories like WordNet 1 and BabelNet 2. With the advent of advanced deep learning models, it is desirous to develop systems that have a good under- standing of languages without such gold standards of word sense. This unsupervised learning can help the model learn better latent representations of words in different contexts. In this paper, we aim to develop a single sys- tem that has knowledge of both multilingual and cross-lingual word sense disambiguation by train- ing models with the combined data for both set- tings. We present our approaches for WSD in the multilingual and cross-lingual domain. The task is treated as a binary-classification problem: whether words have the same sense in the two given pairs of sentences. We experiment with XLM-RoBERTa 1https://wordnet.princeton.edu/ 2https://babelnet.org/ (Conneau et al., 2019), which is based on the Trans- former architecture (Vaswani et al., 2017), as the backbone of our architectures in both the settings. In addition, we also leverage external data and dif- ferent training techniques and data augmentation. The rest of the paper is organized as follows : various related works have been discussed in sec- tion 2, followed by a brief description of the shared task dataset in section 3. The system overview and experimental settings are covered in sections 4 and 5. Sections 6 contain the results. Section 7 con- cludes the paper and also includes scope of future work. 2 Related Work Silberer and Ponzetto (2010) make use of graph algorithms for the word sense disambiguation task. They build a multilingual co-occurence graph in which the multilingual nodes are connected with translation edges and labelled with the tar- get word’s translations as obtained from the corre- sponding contexts. Authors in Banea and Mihalcea (2011), use mul- tilingual vector space which is obtained …

Content truncated to 500 words