Gorynych Transformer at SemEval-2020 Task 6: Multi-task Learning for Definition Extraction

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 59–66 Barcelona, Spain (Online), December 12, 2020. 59 BabelEnconding at SemEval-2020 Task 3: Contextual Similarity as a Combination of Multilingualism and Language Models Lucas R. C. Pessutto1, Tiago de Melo2, Viviane P. Moreira1, Altigran da Silva3 1Institute of Informatics – UFRGS – Brazil; 2Amazonas State University – UEA – Brazil; 3Federal University of Amazonas – UFAM – Brazil lrcpessutto@inf.ufrgs.br, tmelo@uea.edu.br, viviane@inf.ufrgs.br, alti@icomp.ufam.edu.br Abstract This paper describes the system submitted by our team (BabelEnconding) to SemEval-2020 Task 3: Predicting the Graded Effect of Context in Word Similarity. We propose an approach that relies on translation and multilingual language models in order to compute the contextual similarity between pairs of words. Our hypothesis is that evidence from additional languages can leverage the correlation with the human generated scores. BabelEnconding was applied to both subtasks and ranked among the top-3 in six out of eight task/language combinations and was the highest scoring system three times. 1 Introduction Word similarity is a key task in Natural Language Processing (NLP) applications. Language models, such as word embeddings (Mikolov et al., 2013) create vector representations for the words that are able to capture syntactic and semantic relationships. These representations became very popular in the last few years as they have boosted the performance of several NLP tasks. However, since each word is represented by a fixed vector these techniques have problems dealing with polysemous words and identifying subtle meaning changes between different sentences. On the other hand, state-of-the-art language models, like BERT (Devlin et al., 2019) provide a contextualized word representation – the representation of a word relies on its context, which means that the same word may have different representations through the sentences. Thus, BERT models are more suitable for handling polysemous words. Task 3 in SemEval 2020 – Predicting the Graded Effect of Context in Word Similarity (Armendariz et al., 2020a) was motivated by this improvement on language models. The task aims at the design of a similarity measure which captures the human perception of the meaning of words. For that purpose, task organizers built and annotated datasets in four languages – English, Croatian, Finnish, and Slovenian. Each entry in a dataset consists of two target words and two contexts, where each one is a piece of text containing both target words. The global task is divided into two subtasks: 1) predicting the change in the human annotator’s scores of similarity when presented with the same pair of words within two different contexts; and 2) predicting the human scores of similarity for a pair of words within two different contexts. In this paper, we describe BabelEnconding, an approach that relies on machine translation and multi- lingual language models to evaluate the contextual similarity of pairs of words. Our hypothesis is that having similarity information from more languages helps decide on how similar the words are. Considering the eight combinations of language/subtask, BabelEnconding was ranked among the top-3 competitors six times, and was the top …

Content truncated to 500 words