Morphological Priors for Probabilistic Neural Word Embeddings

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 490–500, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Morphological Priors for Probabilistic Neural Word Embeddings Parminder Bhatia∗ Yik Yak, Inc. 3525 Piedmont Rd NE, Building 6, Suite 500 Atlanta, GA parminder@yikyakapp.com Robert Guthrie∗and Jacob Eisenstein School of Interactive Computing Georgia Institute of Technology Atlanta, GA 30312 USA {rguthrie3 + jacobe}@gatech.edu Abstract Word embeddings allow natural language pro- cessing systems to share statistical information across related words. These embeddings are typically based on distributional statistics, mak- ing it difficult for them to generalize to rare or unseen words. We propose to improve word embeddings by incorporating morphological information, capturing shared sub-word fea- tures. Unlike previous work that constructs word embeddings directly from morphemes, we combine morphological and distributional information in a unified probabilistic frame- work, in which the word embedding is a latent variable. The morphological information pro- vides a prior distribution on the latent word em- beddings, which in turn condition a likelihood function over an observed corpus. This ap- proach yields improvements on intrinsic word similarity evaluations, and also in the down- stream task of part-of-speech tagging. 1 Introduction Word embeddings have been shown to improve many natural language processing applications, from lan- guage models (Mikolov et al., 2010) to information extraction (Collobert and Weston, 2008), and from parsing (Chen and Manning, 2014) to machine trans- lation (Cho et al., 2014). Word embeddings leverage a classical idea in natural language processing: use distributional statistics from large amounts of unla- beled data to learn representations that allow sharing ∗The first two authors contributed equally. Code is available at https://github.com/rguthrie3/ MorphologicalPriorsForWordEmbeddings. across related words (Brown et al., 1992). While this approach is undeniably effective, the long-tail nature of linguistic data ensures that there will always be words that are not observed in even the largest cor- pus (Zipf, 1949). There will be many other words which are observed only a handful of times, making the distributional statistics too sparse to accurately estimate the 100- or 1000-dimensional dense vectors that are typically used for word embeddings. These problems are particularly acute in morphologically rich languages like German and Turkish, where each word may have dozens of possible inflections. Recent work has proposed to address this issue by replacing word-level embeddings with embeddings based on subword units: morphemes (Luong et al., 2013; Botha and Blunsom, 2014) or individual char- acters (Santos and Zadrozny, 2014; Ling et al., 2015; Kim et al., 2016). Such models leverage the fact that word meaning is often compositional, arising from subword components. By learning representations of subword units, it is possible to generalize to rare and unseen words. But while morphology and orthography are some- times a signal of semantics, there are also many cases similar spellings do not imply similar meanings: bet- ter-batter, melon-felon, dessert-desert, etc. If each word’s embedding is constrained to be a determinis- tic function of its characters, as in prior work, then it will be difficult to …

Content truncated to 500 words