Few-Shot Semantic Parsing for New Predicates

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 125–137 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 125 Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection Jens Kaiser∗, Sinan Kurtyigit∗, Serge Kotchourko∗, Dominik Schlechtweg Institute for Natural Language Processing, University of Stuttgart {jens.kaiser,sinan.kurtyigit,serge.kotchourko,schlecdk}@ims.uni-stuttgart.de Abstract Lexical semantic change detection is a new and innovative research field. The optimal fine-tuning of models including pre- and post- processing is largely unclear. We optimize existing models by (i) pre-training on large corpora and refining on diachronic target cor- pora tackling the notorious small data problem, and (ii) applying post-processing transforma- tions that have been shown to improve perfor- mance on synchronic tasks. Our results pro- vide a guide for the application and optimiza- tion of lexical semantic change detection mod- els across various learning scenarios. 1 Introduction In recent years Lexical Semantic Change Detec- tion (LSCD), i.e. the detection of word meaning change over time, has seen considerable develop- ments (Tahmasebi et al., 2018; Kutuzov et al., 2018; Hengchen et al., 2021). The recent publication of multi-lingual human-annotated evaluation data from SemEval-2020 Task 1 (Schlechtweg et al., 2020) makes it now possible to compare LSCD models in a variety of scenarios. The task shows a clear dominance of type-based embeddings, al- though these are strongly influenced by the size of training corpora. In order to mitigate this problem we propose pre-training models on large corpora and refine them on diachronic target corpora. We further improve the obtained embeddings with sev- eral post-processing transformations which have been shown to have positive effects on performance in semantic similarity and analogy tasks (Mu et al., 2017; Artetxe et al., 2018b; Raunak et al., 2019) as well as term extraction (H¨atty et al., 2020). Ex- tensive experiments are performed on the German and English LSCD datasets from SemEval-2020 ∗Authors contributed equally, and their ordering was de- termined randomly. Task 1. According to our findings, pre-training is advisable when the target corpora are small and should be done using diachronic data. We further show that pre-training on large corpora strongly interacts with vector dimensionality and propose a simple solution to avoid drastic performance drops. Post-processing often yields further improvements. However, it is hard to find a reliable parameter that performs well across the board. Our experiments suggest that it is possible to use simple pre- and post-processing techniques to improve the state-of- the-art in LSCD. 2 Related Work As evident in Schlechtweg et al. (2020) the field of LSCD is currently dominated by Vector Space Models (VSMs), which can be divided into type- based (static) (Turney and Pantel, 2010) and token-based (contextualized) (Sch¨utze, 1998) mod- els. Prominent type-based models include low- dimensional embeddings such as Global Vectors (GloVe, Pennington et al., 2014) and Skip-Gram with Negative Sampling (SGNS, Mikolov et al., 2013a,b). However, as these models come with the deficiency that they aggregate all senses of a word into a single representation, token-based em- …

Content truncated to 500 words