KEIS@JUST at SemEval-2020 Task 12: Identifying Multilingual Offensive Tweets Using Weighted Ensemble and Fine-Tuned BERT

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 214–221 Barcelona, Spain (Online), December 12, 2020. 214 SST-BERT at SemEval-2020 Task 1: Semantic Shift Tracing by Clustering in BERT-based Embedding Spaces Vani Kanjirangat Sandra Mitrovi´c IDSIA, Lugano, Switzerland {vanik, sandra, alessandro, fabio.rinaldi}@idsia.ch Alessandro Antonucci Fabio Rinaldi Abstract Lexical semantic change detection (also known as semantic shift tracing) is a task of identifying words that have changed their meaning over time. Unsupervised semantic shift tracing, focal point of SemEval2020, is particularly challenging. Given the unsupervised setup, in this work, we propose to identify clusters among different occurrences of each target word, considering these as representatives of different word meanings. As such, disagreements in obtained clusters naturally allow to quantify the level of semantic shift per each target word in four target languages. To leverage this idea, clustering is performed on contextualized (BERT-based) embeddings of word occurrences. The obtained results show that our approach performs well both measured separately (per language) and overall, where we surpass all provided SemEval baselines. 1 Problem Setup Consider two corpora C1 and C2 for a same language but associated with different time stamps (say, respectively, t1 and t2 > t1). Let W be a set of target words occurring in both corpora. Each target word w ∈ W might assume multiple meanings, to be called senses, within the two corpora. A pool of experts annotated a representative amount of occurrences with their corresponding senses. The problem we consider is to characterize the semantic shift related to those senses from one corpus to the other without having access to the expert annotations. In particular, we address the two following two subtasks: • Subtask 1: Decide, for each w ∈ W, whether or not w gained or lost at least a sense between t1 and t2. This is a binary decision task. We will denote this subtask as (S1). • Subtask 2: Define, for the elements of W, a measure of their degree of lexical semantic change between t1 and t2 and sort these elements consequently. This is a ranking task. This subtask will be referred to as (S2). We1 describe two different methods able to address both subtasks. As both methods require a preprocessing step based on transformers, let us start from this preliminary operation. 2 Preprocessing For the proposed approach, we used embeddings derived from BERT (Bidirectional Encoder Represen- tations from Transformers) (Devlin et al., 2018) model to represent the text information in the corpora. BERT uses attention mechanism to learn the contextual relations and reads the input bidirectionally. It is an encoder-only model (as the goal is to generate a language model) opposed to transformers (encoder- decoder model)(Vaswani et al., 2017). BERT is trained on two main objectives, masked language model (MLM) and next sentence prediction (NSP). The corpus data is initially segmented at sentence-level and the BERT Word Piece tokenizer is applied on This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1Our team name in …

Content truncated to 500 words