Warren at SemEval-2020 Task 4: ALBERT and Multi-Task Learning for Commonsense Validation

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 74–80 Barcelona, Spain (Online), December 12, 2020. 74 Random at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering Pierluigi Cassotti University of Bari, Italy pierluigi.cassotti@uniba.it Annalina Caputo ADAPT Centre, Dublin City University, Ireland annalina.caputo@dcu.ie Marco Polignano University of Bari, Italy marco.polignano@uniba.it Pierpaolo Basile University of Bari, Italy pierpaolo.basile@uniba.it Abstract This paper describes the system proposed by the Random team for SemEval-2020 Task 1: Unsu- pervised Lexical Semantic Change Detection. We focus our approach on the detection problem. Given the semantics of words captured by temporal word embeddings in different time periods, we investigate the use of unsupervised methods to detect when the target word has gained or lost senses. To this end, we define a new algorithm based on Gaussian Mixture Models to cluster the target similarities computed over the two periods. We compare the proposed approach with a number of similarity-based thresholds. We found that, although the performance of the detection methods varies across the word embedding algorithms, the combination of Gaussian Mixture with Temporal Referencing resulted in our best system. 1 Introduction The recent development in word embeddings, and their increasing capability to capture lexical semantics has inspired the application of these methods to new tasks and introduced new challenges. The diachronic analysis of language is one of these linguistic tasks which has benefited from the advantages of these new methods, i.e. the capability to build semantic representations of words by skimming through large corpora spanning multiple time periods. SemEval 2020 Task 1 (Schlechtweg et al., 2020) addresses the current lack of a systematic approach for the evaluation of automatic methods for diachronic analysis by proposing a common evaluation framework that comprises two tasks and covers corpora written in four different languages, namely German (Zeitung, 2018; Textarchiv, 2017), English (Alatrash et al., 2020), Latin (McGillivray and Kilgarriff, 2013), and Swedish (Borin et al., 2012). Given two corpora C1 and C2 for two periods t1 and t2, Subtask 1 requires participants to classify a set of target words in two categories: words that have lost or gained senses from t1 to t2 and words that did not, while Subtask 2 requires participants to rank the target words according to their degree of lexical semantic change between the two periods. We tackle the problem of automatically detecting lexical semantic changes with approaches that rely on temporal word embeddings. These approaches create a word vector representation for each time period by exploiting a shared semantic space. Similarity measures can then be used to capture the extent of a word semantic change between two periods. Some temporal word embedding techniques adopt a two-step approach, where they first learn separate word embeddings for each time period and then align the word vectors across multiple time periods (Hamilton et al., 2016). Other dynamic approaches incorporate the alignment directly into the learning stage via the optimisation function (Tahmasebi et al., 2018). Dynamic word embeddings can be further categorised according to the constraint imposed on …

Content truncated to 500 words