TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 771–779 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 771 TransWiC at SemEval-2021 Task 2: Transformer-based Multilingual and Cross-lingual Word-in-Context Disambiguation Hansi Hettiarachchi♥, Tharindu Ranasinghe§ ♥School of Computing and Digital Technology, Birmingham City University, UK §Research Group in Computational Linguistics, University of Wolverhampton, UK hansi.hettiarachchi@mail.bcu.ac.uk tharindu.ranasinghe@wlv.ac.uk Abstract Identifying whether a word carries the same meaning or different meaning in two con- texts is an important research area in natu- ral language processing which plays a signif- icant role in many applications such as ques- tion answering, document summarisation, in- formation retrieval and information extraction. Most of the previous work in this area rely on language-specific resources making it difficult to generalise across languages. Considering this limitation, our approach to SemEval-2021 Task 2 is based only on pretrained transformer models and does not use any language-specific processing and resources. Despite that, our best model achieves 0.90 accuracy for English- English subtask which is very compatible com- pared to the best result of the subtask; 0.93 accuracy. Our approach also achieves satis- factory results in other monolingual and cross- lingual language pairs as well. 1 Introduction Words’ semantics have a dynamic nature which depends on the surrounding context (Pilehvar and Camacho-Collados, 2019). Therefore, the majority of words tends to be polysemous (i.e. have mul- tiple senses). For few examples, words such as "cell", "bank" and "report" can be mentioned. Due to this nature in natural language, it is important to focus on word-in-context sense while extract- ing the meaning of a word which appeared in a text segment. Also, this is a critical requirement to many applications such as question answering, document summarisation, information retrieval and information extraction. Word Sense Disambiguation (WSD)-based ap- proaches were widely used by previous research to tackle this problem (Loureiro and Jorge, 2019; Scarlini et al., 2020). WSD associates the word in a text with its correct meaning from a prede- fined sense inventory (Navigli, 2009). As such inventories, WordNet (Miller, 1995) and Babel- Net (Navigli and Ponzetto, 2012) were commonly used. However, these approaches fail to generalise into different languages as these inventories are of- ten limited to high resource languages. Targeting this gap, SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation is designed to capture the word sense without rely- ing on fixed sense inventories in both monolingual and cross-lingual setting. In summary, this task is designed as a binary classification problem which predicts whether the target word has the same mean- ing or different meaning in different contexts of the same language (monolingual setting) or different languages (cross-lingual setting). This paper describes our submission to SemEval- 2021 Task 2 (Martelli et al., 2021). Our approach is mainly focused on transformer-based models with different text pair classification architectures. We remodel the default text pair classification archi- tecture and introduce several strategies that outper- form the default text pair classification architecture for this task. For effortless generalisation across the languages, we do …

Content truncated to 500 words