JUST at SemEval-2020 Task 11: Detecting Propaganda Techniques Using BERT Pre-trained Model

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 187–193 Barcelona, Spain (Online), December 12, 2020. 187 CMCE at SemEval-2020 Task 1: Clustering on Manifolds of Contextualized Embeddings to Detect Historical Meaning Shifts David Rother1, Thomas Haider2,3, Steffen Eger1 1Department of Computer Science, Technical University of Darmstadt 2Department of Language and Literature, Max Planck Institute for Empirical Aesthetics 3Institut f¨ur Maschinelle Sprachverarbeitung, University of Stuttgart david.rother@stud.tu-darmstadt.de, thomas.haider@ae.mpg.de eger@aiphes.tu-darmstadt.de Abstract This paper describes the system Clustering on Manifolds of Contextualized Embeddings (CMCE) submitted to the SemEval-2020 Task 1 on Unsupervised Lexical Semantic Change Detection. Subtask 1 asks to identify whether or not a word gained/lost a sense across two time periods. Subtask 2 is about computing a ranking of words according to the amount of change their senses underwent. Our system uses contextualized word embeddings from MBERT, whose dimensionality we reduce with an autoencoder and the UMAP algorithm, to be able to use a wider array of clustering algorithms that can automatically determine the number of clusters. We use Hierarchical Density Based Clustering (HDBSCAN) and compare it to Gaussian Mixture Models (GMMs) and other clustering algorithms. Remarkably, with only 10 dimensional MBERT embeddings (reduced from the original size of 768), our submitted model performs best on subtask 1 for English and ranks third in subtask 2 for English. In addition to describing our system, we discuss our hyperparameter configurations and examine why our system lags behind for the other languages involved in the shared task (German, Swedish, Latin). Our code is available at https://github.com/DavidRother/semeval2020-task1 1 Introduction With the advent of increasingly sophisticated distributional models, semantic change analysis has gained considerable popularity in natural language processing. Earlier methods use static word embeddings to identify laws of semantic change (Hamilton et al., 2016a; Hamilton et al., 2016b; Eger and Mehler, 2016). Recently, contextualized embeddings (Devlin et al., 2018) have allowed to track more fine-grained sense changes of words over time (Hu et al., 2019). The SemEval-2020 “Unsupervised Lexical Semantic Change Detection task” (Schlechtweg et al., 2020) aims at two levels of granularity to detect lexical semantic change: 1) a binary classification task to identify whether or not a word lost/gained a sense between two time periods, and 2) a ranking task, where a list of words has to be ordered by the amount of change these have undergone between the same two time periods. Both subtasks have to be solved for four languages: English, German, Latin and Swedish. To address the tasks, we deploy a language-agnostic system with three ingredients. i) We represent words by multilingual contextualized word embeddings (MBERT) (Devlin et al., 2018). ii) We apply dimensionality reduction on the contextualized word embeddings with an autoencoder and the UMAP algorithm (a type of manifold learning), as it is much easier and more efficient to work with low dimensional vectors in subsequent steps, allowing the application of a wider range of clustering algorithms. iii) We employ a hierarchical clustering approach to find potential ‘sense clusters’ for each word. These sense clusters can …

Content truncated to 500 words