CitiusNLP at SemEval-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 275–280 Barcelona, Spain (Online), December 12, 2020. 275 CitiusNLP at SemEval-2020 Task 3: Comparing two Approaches for Word Vector Contextualization Pablo Gamallo Centro Singular de Investigaci´on en Tecnolox´ıas da Informaci´on (CiTIUS) Universidade de Santiago de Compostela, Galiza pablo.gamallo@usc.es Abstract This article describes some unsupervised strategies submitted to SemEval 2020 Task 3, a task which consists of considering the effect of context to compute word similarity. More precisely, given two words in context, the system must predict the degree of similarity of those words considering the context in which they occur, and the system score is compared against human prediction. We compare one approach based on pre-trained BERT models with other strategy relying on static word embeddings and syntactic dependencies. The BERT-based method clearly outperformed the dependency-based strategy. 1 Introduction The goal of SemEval-2020 Task 3 - Predicting the (Graded) Effect of Context in Word Similarity (Armendariz et al., 2020a), is to predict the degree of similarity of two words considering the context in which those words appear. It should be noted that the context of a word will always have very subtle and continuous (graduated) effects on the lexical meaning, even in the case of those words that are not considered polysemous (Armendariz et al., 2020b). This view of lexical meaning variation as a continuum, inspired in the notion of sense discrimination by Sch¨utze (1998), is quite different from resource-based word sense disambiguation approaches (Navigli and Ponzetto, 2012), which use the local context to predict discrete changes in meaning: the different predefined senses of a polysemous word. In order to grasp the contextualized sense of words, we will make use of two different strategies: Transformers: Contextualized word representations produced by BERT, a bi-directional transformer- based language model for context-sensitive word representations (Devlin et al., 2019). Dependencies: Contextualized word representations produced by considering the static embeddings of syntactically related words to each target word. This paper is organized as follows. The two strategies are described in Section 2. Experiments and results are presented in Section 3. Finally, conclusions are addressed in Section 4. 2 Two Methods for Word Contextualization As mentioned in the introduction, we made use of two different strategies, Transformers and Dependencies, to tackle the challenge of contextualizing word senses. 2.1 Transformers and BERT The first strategy is based on a specific Transformer implementation, namely BERT: Bidirectional Encoder Representations from Transformers (Devlin et al., 2018). Transformers are able to consider word context thanks to the self-attention mechanism, whose objective is to take into account specific parts of the input This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 276 sequence while processing a word. Self-attention allows Transformers to deal with distant contexts of target words, being this mechanism the main difference with regard to Recurrent Neural Networks. BERT is a bi-directional transformer-based language model that can either be used to extract high quality language features from input text or can be fine-tuned …

Content truncated to 500 words