Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 724–729 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 724 Lotus at SemEval-2021 Task 2: Combination of BERT and Paraphrasing for English Word Sense Disambiguation Niloofar Ranjbar Department of Computer Engineering Jam School of Engineering Persian Gulf University nranjbar@pgu.ac.ir Hossein Zeinali Department of Computer Engineering Amirkabir University of Technology hzeinali@aut.ac.ir Abstract In this paper, we describe our proposed meth- ods for the multilingual word-in-Context dis- ambiguation task in SemEval-2021. In this task, systems should determine whether a word that occurs in two different sentences is used with the same meaning or not. We proposed several methods using a pre-trained BERT model. In two of them, we paraphrased sentences and add them as input to the BERT, and in one of them, we used WordNet to add some extra lexical information. We evaluated our proposed methods on test data in SemEval- 2021 task 2. 1 Introduction Measuring semantic similarity is a task that is im- portant in many applications such as summariza- tion, plagiarism detection, etc. To measure the se- mantic similarity between two words or sentences, the words’ meaning in their contexts should be understood. In “Task 2: Multilingual and Cross- lingual Word-in-Context Disambiguation” of Se- mEval 2021 (Martelli et al., 2021), systems should determine whether a word that occurs in two differ- ent sentences is used with the same meaning or not. The considered languages in the Word-in-Context Disambiguation task of the evaluation are Arabic, Chinese, English, French, and Russian (Martelli et al., 2021). Due to our internal limitation, we only evaluated the proposed methods on the En- glish dataset. In most cases, humans can understand the cor- rect meaning of each word by paying attention to the context of that word. This was the main reason for proposing the attention mechanism for machine translation (Vaswani et al., 2017). Because atten- tion plays an important role in BERT topology (De- vlin et al., 2019), we believe it should work for the WSD task. However, in some cases, the two sentences may be about the same subject, while the target word has a different meaning. In these cases, find out that whether the target word has the same meaning in both sentences or not is difficult. To overcome this challenge, we need to add some extra information about the contexts to the input of the BERT. We proposed four main methods for this task. In the first method, we use the BERT model as a language representation model in which the inputs are the two sentences that come in a row and are separated by a “[SEP]” token. Furthermore, the target word is surrounded by a “[TGT]” token in both sentences. Then, the first output of the last layer is used as a classifier to determine whether the word has the same meaning in the two sentences or not. In the second one, we added some information about the target word from the WordNet1 to the …

Content truncated to 500 words