MIDAS at SemEval-2020 Task 10: Emphasis Selection Using Label Distribution Learning and Contextual Embeddings

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 180–186 Barcelona, Spain (Online), December 12, 2020. 180 CIRCE at SemEval-2020 Task 1: Ensembling Context-Free and Context-Dependent Word Representations Martin P¨omsl∗ Osnabr¨uck University mpoemsl@uos.de Roman Lyapin Cogent Labs Inc. rlyapin@cogent.co.jp Abstract This paper describes the winning contribution to SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection (Subtask 2) handed in by team UG Student Intern. We present an ensemble model that makes predictions based on context-free and context-dependent word representations. The key findings are that (1) context-free word representations are a powerful and robust baseline, (2) a sentence classification objective can be used to obtain useful context- dependent word representations, and (3) combining those representations increases performance on some datasets while decreasing performance on others. 1 Introduction SemEval-2020 Task 1 poses an evaluation framework for unsupervised Lexical Semantic Change Detection (LSCD). Its two subtasks operate on a set of non-parallel corpus pairs from two different time periods and are evaluated against human annotations for semantic change of a subset of words. Subtask 1 requires binary classification of whether or not the meaning of the words has changed. Subtask 2 requires ranking the words by degree of lexical semantic change and is evaluated in Spearman’s rank-order correlation coefficient ρ (Schlechtweg et al., 2020). This paper primarily addresses Subtask 2. One of the most successful methods for predicting the degree of semantic change of words is comparing context-free (static) semantic vector spaces. Such models seperately induce word vectors for all words in two corpora (e.g. with Word2Vec (Mikolov et al., 2013a)), align the resulting vector spaces and take the distance of the word vectors as a measure of semantic change (Schlechtweg et al., 2019). With recent advances in language model pretraining, it is now possible to extract context-dependent (contextualized) word representations for each use of a word in the two corpora by using a language model (e.g. BERT (Devlin et al., 2019)) as feature extractor. The distance between these context-dependent word representations can then be taken as a measure of semantic change (Giulianelli, 2019). In this paper, we present a model based on context-free word representations, a model based on context- dependent word representations and an ensemble model that combines their predictions. We obtain context-free representations by following the methodology of the best model reported by Schlechtweg et al. (2019), aligning SGNS vectors. For context-dependent representations, we finetune BERT with a sentence classification objective - predicting the time period of sentences - and extract internal representations for all words from the finetuned BERT model. We show that this classification finetuning can both produce useful word representations and provide an indicator for how to parameterize the ensemble. In the evaluation phase of SemEval-2020 Task 1, the context-free model ranked first out of 128 contributions to Subtask 2. The ensemble model performed better in one language, but significantly worse in another, causing it to be ranked fifth among all contributions. The context-dependent model ranked 69th, suggesting that it can sometimes add information to the …

Content truncated to 500 words