GX at SemEval-2021 Task 2: BERT with Lemma Information for MCL-WiC Task

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 706–712 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 706 GX at SemEval-2021 Task 2: BERT with Lemma Information for MCL-WiC Task Wanying Xie Beijing Language and Culture University, China xiewanying07@gmail.com Abstract This paper presents the GX system for the Mul- tilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC) task. The pur- pose of the MCL-WiC task is to tackle the challenge of capturing the polysemous nature of words without relying on a fixed sense in- ventory in a multilingual and cross-lingual set- ting. To solve the problems, we use context- specific word embeddings from BERT to elim- inate the ambiguity between words in different contexts. For languages without an available training corpus, such as Chinese, we use neu- ral machine translation model to translate the English data released by the organizers to ob- tain available pseudo-data. In this paper, we apply our system to the English and Chinese multilingual setting and the experimental re- sults show that our method has certain advan- tages.1 1 Introduction In recent years, contextual embeddings have drawn much attention. The approaches of calculating contextual embeddings include multi-prototype embeddings, sense-based and contextualized em- beddings (Camacho-Collados and Pilehvar, 2018). However, it is not easy to evaluate such multiple embedding methods in one framework. Pilehvar and Camacho-Collados (2019) present a large-scale word in context dataset to focus on the dynamic se- mantics of words. Following and expanding them, the MCL-WiC task (Martelli et al., 2021) performs a binary classification task that indicates whether the target word is used with the same or different meanings in the same language (multilingual data set) or across different languages (cross-lingual data set). Besides, it is the first SemEval task for Word-in-Context disambiguation (Martelli et al., 2021). 1Reproducible code: https://github.com/yingwaner/bert4wic A typical solution to the problems is obtain- ing context-specific word embeddings, such as Context2vec (Melamud et al., 2016) and Bidirec- tional Encoder Representations from Transformers (BERT) (Devlin et al., 2019). BERT is designed to pre-train deep bidirectional representation in unla- beled texts by jointly conditioning in the left and right contexts of all layers. Due to its powerful capabilities and easy deployment, we use BERT as our major system and fine-tune on the training data released by the organizer to get the context-specific word embeddings. In this paper, we participate in the sub-task of multilingual settings in English and Chinese. The organizer only provides English training data, and we fine-tune the pre-trained English BERT model based on this data. For Chinese tasks where no training set is available, we train a satisfactory neural machine translation (NMT) model to trans- late the English training set into Chinese and then fine-tune the Chinese BERT model based on the pseudo-data. The experimental results show that our method achieves 82.7% in English multilingual setting and 76.7% in Chinese multilingual setting. 2 Background In this section, we will briefly introduce the word- in-context task and the structure of BERT for the sentence pair classification …

Content truncated to 500 words