Hitachi at SemEval-2020 Task 8: Simple but Effective Modality Ensemble for Meme Emotion Recognition

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 126–134 Barcelona, Spain (Online), December 12, 2020. 126 UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection Andrey Kutuzov University of Oslo, Norway andreku@ifi.uio.no Mario Giulianelli University of Amsterdam, Netherlands m.giulianelli@uva.nl Abstract We apply contextualised word embeddings to lexical semantic change detection in the SemEval- 2020 Shared Task 1. This paper focuses on Subtask 2, ranking words by the degree of their semantic drift over time. We analyse the performance of two contextualising architectures (BERT and ELMo) and three change detection algorithms. We find that the most effective algorithms rely on the cosine similarity between averaged token embeddings and the pairwise distances between token embeddings. They outperform strong baselines by a large margin (in the post-evaluation phase, we have the best Subtask 2 submission for SemEval-2020 Task 1), but interestingly, the choice of a particular algorithm depends on the distribution of gold scores in the test set. 1 Introduction Words change their meaning over time in all natural languages. The emergence of large representative historical corpora and powerful data-driven semantic models has allowed researchers to track meaning change. SemEval-2020 Shared Task 1 challenges its participants to classify a list of target words into stable or changed (Subtask 1) and/or to rank these words by the degree of their semantic change (Subtask 2) (Schlechtweg et al., 2020). The task is multilingual: it includes four lists of target words, respectively for English, German, Latin, and Swedish (several dozen words each). Each word list is accompanied with two historical corpora of varying size, consisting of texts created in two different time periods. The shared task organisers additionally provided frequency-based and distributional baseline methods. We participated in Subtask 21 as the UiO-UvA team and investigated the potential of contextualised embeddings for the detection of lexical semantic change. Our systems are based on ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) language models, and employ 3 different algorithms to compare contextualised embeddings diachronically. Our evaluation phase submission to the shared task ranked 9th out of 34 participating teams, while our post-evaluation phase submission is the best from those published on the shared task website2 (but some knowledge of the test sets statistics was needed, see below). In this paper, we extensively evaluate all combinations of architectures, training corpora and change detection algorithms, using 5 test sets in 4 languages. Our main findings are twofold: 1) In 3 out of 5 test sets, ELMo consistently outperforms BERT, while being much faster in training and inference; 2) Cosine similarity of averaged contextualised embeddings and average pairwise distance between contextualised embeddings are the two best-performing change detection algorithms, but different test sets show strong preference to either the former or the latter. This preference shows strong correlation with the distribution of gold scores in a test set. While it may indicate that there is a bias in the available test sets, this finding remains yet unexplained. Our implementations of all the evaluated …

Content truncated to 500 words