PALI at SemEval-2021 Task 2: Fine-Tune XLM-RoBERTa for Word in Context Disambiguation

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 713–718 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 713 PALI at SemEval-2021 Task 2: Fine-Tune XLM-RoBERTa for Word in Context Disambiguation Shuyi Xie, Jian Ma, Haiqin Yang§, Lianxin Jiang, Yang Mo, and Jianping Shen Ping An Life Insurance, Ltd. Shenzhen, Guangdong province, China {XIESHUYI542, MAJIAN446, JIANGLIANXIN769, MOYANG853, SHENJIANPING324}@pingan.com.cn § the corresponding author, email: hqyang@ieee.org Abstract This paper presents the PALI team’s winning system for SemEval-2021 Task 2: Multilin- gual and Cross-lingual Word-in-Context Dis- ambiguation. We fine-tune XLM-RoBERTa model to solve the task of word in context dis- ambiguation, i.e., to determine whether the tar- get word in the two contexts contains the same meaning or not. In implementation, we first specifically design an input tag to emphasize the target word in the contexts. Second, we construct a new vector on the fine-tuned em- beddings from XLM-RoBERTa and feed it to a fully-connected network to output the prob- ability of whether the target word in the con- text has the same meaning or not. The new vector is attained by concatenating the embed- ding of the [CLS] token and the embeddings of the target word in the contexts. In training, we explore several tricks, such as the Ranger optimizer, data augmentation, and adversar- ial training, to improve the model prediction. Consequently, we attain the first place in all four cross-lingual tasks. 1 Introduction This year, the SemEval-2021 task 2, multilingual and cross-lingual word-in-context (WiC) disam- biguation (Martelli et al., 2021), defines the task of identifying the polysemous nature of words with- out relying on a fixed sense inventory in a multi- lingual and cross-lingual setting. The task aims to perform a binary classification task to determine whether the target word contains the same meaning or not in two given contexts under both the same language (multilingual) setting and the different languages (cross-lingual) setting. In the multilin- gual setting, the tasks consist of English-English (En-En), Arabic-Arabic (Ar-Ar), French-French (Fr-Fr), Chinese-Chinese (Zh-Zh) and Russian- Russian (Ru-Ru) while in the cross-lingual set- ting, the tasks consist of English-Chinese (En-Zh), English-French (En-Fr), English-Russian (En-Ru), and English-Arabic (En-Ar). The tasks contain the following challenges: • The same word may deliver different mean- ings in different context (Lei et al., 2021). • The training data is scarce. For example, in the multilingual tasks, there is only training data in En-En, while in the cross-lingual tasks, there is no training data. To overcome these challenges, we explore the uniqueness of the tasks and implement several key technologies: • First, we follow (Botha et al., 2020) to spe- cially design an input tag for the multilingual pre-training XLM-RoBERTa model to empha- size the target word in the contexts. That is, the target word is encompassed by the special symbols of <t> and </t>. Meanwhile, the given two contexts are concatenated by the <SEP> token. • Second, we apply data augmentation and add external data from WordNet to enrich the train- ing data. It is …

Content truncated to 500 words