Learning Word Meta-Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1351–1360, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Learning Word Meta-Embeddings Wenpeng Yin and Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany wenpeng@cis.lmu.de Abstract Word embeddings – distributed represen- tations of words – in deep learning are beneficial for many tasks in NLP. How- ever, different embedding sets vary greatly in quality and characteristics of the cap- tured information. Instead of relying on a more advanced algorithm for embed- ding learning, this paper proposes an en- semble approach of combining different public embedding sets with the aim of learning metaembeddings. Experiments on word similarity and analogy tasks and on part-of-speech tagging show better per- formance of metaembeddings compared to individual embedding sets. One advan- tage of metaembeddings is the increased vocabulary coverage. We release our metaembeddings publicly at http:// cistern.cis.lmu.de/meta-emb. 1 Introduction Recently, deep neural network (NN) models have achieved remarkable results in NLP (Collobert and Weston, 2008; Sutskever et al., 2014; Yin and Sch¨utze, 2015). One reason for these results are word embeddings, compact distributed word representations learned in an unsupervised manner from large corpora (Bengio et al., 2003; Mnih and Hinton, 2009; Mikolov et al., 2013a; Pennington et al., 2014). Some prior work has studied differences in per- formance of different embedding sets. For exam- ple, Chen et al. (2013) show that the embedding sets HLBL (Mnih and Hinton, 2009), SENNA (Collobert and Weston, 2008), Turian (Turian et al., 2010) and Huang (Huang et al., 2012) have great variance in quality and characteristics of the semantics captured. Hill et al. (2014; 2015a) show that embeddings learned by NN machine transla- tion models can outperform three representative monolingual embedding sets: word2vec (Mikolov et al., 2013b), GloVe (Pennington et al., 2014) and CW (Collobert and Weston, 2008). Bansal et al. (2014) find that Brown clustering, SENNA, CW, Huang and word2vec yield significant gains for dependency parsing. Moreover, using these repre- sentations together achieved the best results, sug- gesting their complementarity. These prior stud- ies motivate us to explore an ensemble approach. Each embedding set is trained by a different NN on a different corpus, hence can be treated as a distinct description of words. We want to lever- age this diversity to learn better-performing word embeddings. Our expectation is that the ensemble contains more information than each component embedding set. The ensemble approach has two benefits. First, enhancement of the representations: metaembed- dings perform better than the individual embed- ding sets. Second, coverage: metaembeddings cover more words than the individual embedding sets. The first three ensemble methods we intro- duce are CONC, SVD and 1TON and they directly only have the benefit of enhancement. They learn metaembeddings on the overlapping vocabulary of the embedding sets. CONC concatenates the vec- tors of a word from the different embedding sets. SVD performs dimension reduction on this con- catenation. 1TON assumes that a metaembedding for the word exists, e.g., it can be a randomly …

Content truncated to 500 words