DuluthNLP at SemEval-2021 Task 7: Fine-Tuning RoBERTa Model for Humor Detection and Offense Rating

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 157–162 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 157 SkoltechNLP at SemEval-2021 Task 2: Generating Cross-Lingual Training Data for the Word-in-Context Task Anton Razzhigaev1, Nikolay Arefyev2,3,4, and Alexander Panchenko1 1Skolkovo Institute of Science and Technology, Russia 2Samsung Research Center Russia, Russia 3Lomonosov Moscow State University, Russia 4HSE University, Russia {anton.razzhigaev,a.panchenko}@skoltech.ru narefjev@cs.msu.ru Abstract In this paper, we present a system for the solu- tion of the cross-lingual and multilingual word- in-context disambiguation task. Task organiz- ers provided monolingual data in several lan- guages, but no cross-lingual training data were available. To address the lack of the officially provided cross-lingual training data, we de- cided to generate such data ourselves. We de- scribe a simple yet effective approach based on machine translation and back translation of the lexical units to the original language used in the context of this shared task. In our exper- iments, we used a neural system based on the XLM-R (Conneau et al., 2020), a pre-trained transformer-based masked language model, as a baseline. We show the effectiveness of the proposed approach as it allows to substantially improve the performance of this strong neu- ral baseline model. In addition, in this study, we present multiple types of the XLM-R based classifier, experimenting with various ways of mixing information from the first and second occurrences of the target word in two samples. 1 Introduction The goal of the second task of SemEval-2021 (Martelli et al., 2021) is to perform multilingual and cross-lingual word-in-context disambiguation. More specifically, participants are asked to distin- guish whether the meanings of a target word in two provided contexts are the same or not. Organizers provided a training set of 8 000 English language (en-en) context pairs and validation sets of 1 000 context pairs for English-English (en-en), French- French (fr-fr), Russian-Russian (ru-ru), Arabic- Arabic (ar-ar), and Chinese-Chinese (zh-zh) lan- guages. Since no cross-lingual training data were provided, except for a very small trial set barely usable for training, we decided to venture into gen- erating such data automatically. Essentially, the given task is a binary classifica- tion problem. The first question was which super- vised model to use for the classification of context pairs. Recently, pre-trained masked language mod- els such as BERT (Devlin et al., 2019) and XLM- R (Conneau et al., 2020) have been used to reach promising results in a variety of similar NLU clas- sification tasks. Thus, we decided to make use of contextualized embeddings from XLM-R, which provides multilingual-lingual embeddings for more than 100 languages, covering all language pairs of interest in the shared task. In all our experiments, this model is used as the backbone. A straightforward way of solving tasks where two contexts are to be compared, as the word-in- context tasks, is to use deep contextualized em- beddings and train a classifier over these embed- dings as has been explored in the original monolin- gual word-in-context task (Pilehvar and Camacho- Collados, 2019). …

Content truncated to 500 words