Continuous Word Embedding Fusion via Spectral Decomposition

Proceedings of the 22nd Conference on Computational Natural Language Learning (CoNLL 2018), pages 11–20 Brussels, Belgium, October 31 - November 1, 2018. c⃝2018 Association for Computational Linguistics 11 Continuous Word Embedding Fusion via Spectral Decomposition Tianfan Fu Georgia Institute of Technology Atlanta, GA, USA tfu42@gatech.edu Cheng Zhang Microsoft Research Cambridge Cambridge, CB1 2FB, UK cheng.zhang@microsoft.com Stephan Mandt Los Angeles, CA, USA stephan.mandt@gmail.com Abstract Word embeddings have become a mainstream tool in statistical natural language processing. Practitioners often use pre-trained word vec- tors, which were trained on large generic text corpora, and which are readily available on the web. However, pre-trained word vectors often- times lack important words from specific do- mains. It is therefore often desirable to extend the vocabulary and embed new words into a set of pre-trained word vectors. In this paper, we present an efficient method for including new words from a specialized corpus, contain- ing new words, into pre-trained generic word embeddings. We build on the established view of word embeddings as matrix factorizations to present a spectral algorithm for this task. Ex- periments on several domain-specific corpora with specialized vocabularies demonstrate that our method is able to embed the new words efficiently into the original embedding space. Compared to competing methods, our method is faster, parameter-free, and deterministic. 1 Introduction There has been a recent surge of neural word em- bedding models (Mikolov et al., 2013a,b). These models have been shown to perform well in a va- riety of NLP problems, such as word similarity and relational analogy tasks. Word embeddings play a crucial role in diverse fields such as com- puter vision (Hwang and Sigal, 2014), news clas- sification (Kenter and De Rijke, 2015; Phung and De Vine, 2015), machine translation (Zou et al., 2013; Wu et al., 2014), and have been extended in various ways (Rudolph et al., 2016; Bamler and Mandt, 2017; Peters et al., 2018). Instead of training word embeddings from scratch, practitioners often resort to high-quality, pre-trained word embeddings which can be down- loaded from the web. These embeddings were trained on massive corpora, such as online news. The downside is that their vocabulary is often re- stricted, and by nature, is very generic. An open question remains of how to optimally include new words from highly specialized text corpora into an existing word embedding fit. Such transfer learning has several advantages. First, one saves the com- putational burden of learning high-quality word vectors from scratch. Second, one can already rely on the fact that the majority of word embed- ding vectors are semantically meaningful. Third, as we show in this paper, there are deterministic and parameter-free approaches that fulfill this goal, making the scheme robust and reproducible. For a practical application, imagine that we are given a small corpus, such as a collection of sci- entific articles, and our goal is to include the as- sociated vocabulary into a pre-trained set of word vectors which were learned on Google Books or Wikipedia. The scientific corpus contains both common words …

Content truncated to 500 words