A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2898–2904 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings Wei Yang University of Waterloo w85yang@uwaterloo.ca Wei Lu Singapore University of Technology and Design luwei@sutd.edu.sg Vincent W. Zheng Advanced Digital Sciences Center vincent.zheng @adsc.com.sg Abstract Learning word embeddings has received a significant amount of attention recently. Often, word embeddings are learned in an unsupervised manner from a large collec- tion of text. The genre of the text typi- cally plays an important role in the effec- tiveness of the resulting embeddings. How to effectively train word embedding mod- els using data from different domains re- mains a problem that is underexplored. In this paper, we present a simple yet effec- tive method for learning word embeddings based on text from different domains. We demonstrate the effectiveness of our ap- proach through extensive experiments on various down-stream NLP tasks. 1 Introduction Recently, the learning of distributed representa- tions for natural language words (or word embed- dings) has received a significant amount of atten- tion (Mnih and Hinton, 2007; Turian et al., 2010; Mikolov et al., 2013a,b,c; Pennington et al., 2014). Such representations were shown to be able to cap- ture syntactic and semantic level information asso- ciated with words (Mikolov et al., 2013a). Word embeddings were shown effective in tasks such as named entity recognition (Sienˇcnik, 2015), sen- timent analysis (Li and Lu, 2017) and syntactic parsing (Durrett and Klein, 2015). One common assumption made by most of the embedding meth- ods is that, the text corpus is from one single do- main; e.g., articles from bioinformatics. How- ever, in practice, there are often text corpora from multiple domains; e.g., we may have text collec- tions from broadcast news or Web blogs, whose words are not necessarily limited to bioinformat- ics. Can these corpora from different domains help learn better word embeddings, so as to improve the downstream NLP applications in a target domain like bioinformatics? Our answer is yes, because despite the domain differences, these additional domains do introduce more text data converying useful information (i.e., more words, more word co-occurrences), which can be helpful for consoli- dating the word embeddings in the target bioinfor- matics domain. In this paper, we propose a simple and easy- to-implement approach for learning cross-domain word embeddings. Our model can be seen as a regularized skip-gram model (Mikolov et al., 2013a,b), where the source domain information is selectively incorporated for learning the target do- main word embeddings in a principled manner. 2 Related Work Learning a continuous representation for words has been studied for quite a while (Hinton et al., 1986). Many earlier word embedding meth- ods employed the computationally expensive neu- ral network architectures (Collobert and Weston, 2008; Mikolov et al., 2013c). Recently, an ef- ficient method for learning word representations, namely the skip-gram model (Mikolov et al., 2013a,b) was proposed and implemented in the widely used word2vec toolkit. It …

Content truncated to 500 words