Domain-Specific Word Embeddings with Structure Prediction

Domain-Specific Word Embeddings with Structure Prediction David Lassner1,2∗ Stephanie Brandl1,2,3∗ Anne Baillot4 Shinichi Nakajima1,2,5 1TU Berlin, Germany 2BIFOLD, Germany 3University of Copenhagen, Denmark 4Le Mans Universit´e, France 5RIKEN Center for AIP, Japan {lassner@tu-berlin.de,brandl@di.ku.dk} ∗Authors contributed equally. Abstract Complementary to finding good general word embeddings, an important question for repre- sentation learning is to find dynamic word em- beddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic em- beddings can only be compared after post- alignment. We propose novel word embedding methods that provide general word repre- sentations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding align- ment simultaneously. We present an empiri- cal evaluation on New York Times articles and two English Wikipedia datasets with arti- cles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive. 1 Introduction Word embeddings (Mikolov et al., 2013b; Pennington et al., 2014) are a powerful tool for word-level representation in a vector space that captures semantic and syntactic relations between words. They have been successfully used in many applications such as text classification (Joulin et al., 2016) and machine translation (Mikolov et al., 2013a). Word embeddings highly depend on their training corpus. For example, technical terms used in scientific documents can have a different meaning in other domains, and words can change their meaning over time—‘‘apple’’ did not mean a tech company before Apple Inc. was founded. On the other hand, such local or domain-specific representations are also not in- dependent of each other, because most words are expected to have a similar meaning across domains. There are many situations where a given target corpus is considered to have some structure. For example, when analyzing news articles, one can expect that articles published in 2000 and 2001 are more similar to each other than the ones from 2000 and 2010. When analyzing scientific articles, uses of technical terms are expected to be similar in articles on similar fields of science. This implies that the structure of a corpus can be a useful side resource for obtaining better word representation. Various approaches to analyze semantic shifts in text have been proposed where typically first individual static embeddings are trained and then aligned afterwards (e.g., Kulkarni et al., 2015; Hamilton et al., 2016; Kutuzov et al., 2018; Tahmasebi et al., 2018). As most word embed- dings are invariant with respect to rotation and scaling, it is necessary to map word embeddings from different training procedures into the same vector space in order to compare them. This procedure is …

Content truncated to 500 words