Comparing Contextual and Static Word Embeddings with Small Data

Comparing Contextual and Static Word Embeddings with Small Philosophical Data Wei Zhou Institute for Natural Language Processing University of Stuttgart weizhou14330@gmail.com Jelke Bloem Institute for Logic, Language and Computation University of Amsterdam j.bloem@uva.nl Abstract For domain-specific NLP tasks, applying word embeddings trained on general cor- pora is not optimal. Meanwhile, training domain-specific word representations poses challenges to dataset construction and embed- ding evaluation. In this paper, we present and compare ELMo and Word2Vec models trained/finetuned on philosophical data. For evaluation, a conceptual network was used. Results show that contextualized models pro- vide better word embeddings than static mod- els and that merging embeddings from differ- ent models boosts task performance. 1 Introduction Statistical distributions of terms in context can be used to characterize their semantic behavior (Lenci, 2018). This is the fundamental idea that distribu- tional models of language are built upon. When trained on large corpora, these models can provide valid word representations which can be further utilized in various downstream NLP tasks. Two common models are Word2Vec’s skipgram model (W2V, Mikolov et al., 2013) and the ELMo model (Peters et al., 2018). Although word embeddings pretrained on large corpora provide good mean- ing representations, using them in domain-specific tasks does not achieve good results (Nooralahzadeh et al., 2018). This is because the semantic space of a certain domain can be different from that of general language. For instance, the word substance refers to matter in ordinary language but in philo- sophical contexts it is a technical term from meta- physics pertaining to entities (Robinson, 2020). Contextualized models like ELMo address this problem to some extent, but require a lot of domain- specific data to obtain a tailored model, while such datasets are typically smaller. The main contribu- tions of this paper are: • We trained and finetuned ELMo models with a philosophical corpus. • We examined and compared models with con- textual embeddings and static embeddings with intrinsic evaluations. • We experimented with combining finetuned W2V and pretrained ELMo embeddings for representations of philosophical terms. 2 Related work Efforts have been made in the following direc- tions with regards to creating domain-specific word embeddings. Firstly, the construction of domain- specific corpora. Roy et al. (2017) appended man- ual annotations (predicate-argument structure) to training data in the field of cybersecurity. The addi- tional annotation makes the original dataset more suited to the task of training cybersecurity embed- dings. Secondly, refinement can be carried out on existing embeddings. Boukkouri et al. (2019) combined W2V embeddings trained on a small domain-specific corpus with ELMo embeddings and evaluated them on a clinical entity recognition task. They found combined embeddings outper- formed embeddings trained on large corpora in the medical domain. Lastly, one can also explore suitable models for training with small amount of data. Herbelot and Baroni (2017) proposed a re- fined W2V model called Nonce2Vec (N2V), which learns word meanings from tiny data. The N2V model takes a high-risk learning approach with heightened learning rate and larger window size to process …

Content truncated to 500 words