UoB_UK at SemEval 2021 Task 2: Zero-Shot and Few-Shot Learning for Multi-lingual and Cross-lingual Word Sense Disambiguation.

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 738–742 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 738 UoB UK at SemEval 2021 Task 2: Zero-Shot and Few-Shot Learning for Multi-lingual and Cross-lingual Word Sense Disambiguation. Wei Li, Harish Tayyar Madabushi and Mark Lee School of Computer Science University of Birmingham United Kingdom WXL885@student.bham.ac.uk Harish@HarishTayyarMadabushi.com, M.G.Lee@bham.ac.uk Abstract This paper describes our submission to Se- mEval 2021 Task 2. We compare XLM- RoBERTa Base and Large in the few-shot and zero-shot settings and additionally test the effectiveness of using a k-nearest neighbors classifier in the few-shot setting instead of the more traditional multi-layered perceptron. Our experiments on both the multi-lingual and cross-lingual data show that XLM-RoBERTa Large, unlike the Base version, seems to be able to more effectively transfer learning in a few-shot setting and that the k-nearest neigh- bors classifier is indeed a more powerful classi- fier than a multi-layered perceptron when used in few-shot learning. 1 Introduction and Motivation Word Sense Disambiguation (WSD) is the task of disambiguating semantic meaning at the word level and is an important part of Natural Language Pro- cessing (NLP) with applications in several down- stream tasks (Wang et al., 2020b). In recent years, Few Shot Learning (FSL) has been successful in several domains (Wang et al., 2020a) including in NLP (Yan et al., 2018). Modern deep neural net- works require a significant amount of training data that might not always be available. FSL is a solu- tion to this problem, wherein training data from a related domain or language can be used to augment training on the target domain/language with signifi- cantly less data. FSL can be characterised as n-way k-shot classification. We participate in SemEval Task 2 Multilingual and Cross-lingual Word in Context Disambigua- tion (Martelli et al., 2021), which provides 8,000 training examples in English but only 32 in the other target languages. In addition the tasks re- quires disambiguation of word pairs in the cross lingual setting between EN-AR, EN-FR, EN-RU, and EN-ZH, where only 8 examples each are avail- able. This sparsity in training data led us to explore the use of FSL for this task. We hypothesise that: 1. “Large” models, which have been shown to have a lot more linguistic information and high-level generalisability, are more likely to be able to generalise their learning in the few- shot setting, and 2. that the use of a k-nearest neighbors (KNN) classifier is likely to be more effective in the few-shot setting. This hypothesis is based on our exploration of related work (Section 2). Our experiments on both the multi-lingual and cross-lingual data show that these hypotheses are in fact correct. We show that XLM-RoBERTa Large, unlike the Base version, seems to be able to more effectively transfer knowledge in a few-shot set- ting, and that the KNN classifier is indeed a more powerful classifier than a multi-layered perceptron (MLP) when used in few-shot learning. To ensure reproducibility and so other researchers can …

Content truncated to 500 words