Morphological Smoothing and Extrapolation of Word Embeddings

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1651–1660, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Morphological Smoothing and Extrapolation of Word Embeddings Ryan Cotterell Department of Computer Science Johns Hopkins University, USA ryan.cotterell@jhu.edu Hinrich Sch¨utze CIS LMU Munich, Germany inquiries@cis.lmu.org Jason Eisner Department of Computer Science Johns Hopkins University, USA jason@cs.jhu.edu Abstract Languages with rich inflectional morphol- ogy exhibit lexical data sparsity, since the word used to express a given concept will vary with the syntactic context. For in- stance, each count noun in Czech has 12 forms (where English uses only singular and plural). Even in large corpora, we are un- likely to observe all inflections of a given lemma. This reduces the vocabulary cover- age of methods that induce continuous rep- resentations for words from distributional corpus information. We solve this prob- lem by exploiting existing morphological resources that can enumerate a word’s com- ponent morphemes. We present a latent- variable Gaussian graphical model that al- lows us to extrapolate continuous represen- tations for words not observed in the train- ing corpus, as well as smoothing the repre- sentations provided for the observed words. The latent variables represent embeddings of morphemes, which combine to create em- beddings of words. Over several languages and training sizes, our model improves the embeddings for words, when evaluated on an analogy task, skip-gram predictive accu- racy, and word similarity. 1 Introduction Representations of words as high-dimensional real vectors have been shown to benefit a wide variety of NLP tasks. Because of this demonstrated utility, many aspects of vector representations have been explored recently in the literature. One of the most interesting discoveries is that these representations capture meaningful morpho-syntactic and seman- tic properties through very simple linear relations: in a semantic vector space, we observe that vtalked − vtalk ≈ vdrank − vdrink. (1) That this equation approximately holds across many morphologically related 4-tuples indicates bebieron comieron bebemos comemos Figure 1: A visual depiction of the vector offset method for morpho-syntactic analogies in R2. We expect bebieron and bebemos to have the same relation (vector offset shown as solid vector) as comieron and comemos. that the learned embeddings capture a feature of English morphology—adding the past tense feature roughly corresponds to adding a certain vector. Moreover, manipulating this equation yields what we will call the vector offset method (Mikolov et al., 2013c) for approximating other vectors. For instance, if we only know the vectors for the Spanish words comieron (ate), comemos (eat) and bebieron (drank), we can produce an ap- proximation of the vector for bebemos (drink), as shown in Figure 1. Many languages exhibit much richer morphol- ogy than English. While English nouns com- monly take two forms – singular and plural— Czech nouns take 12 and Turkish nouns take over 30. This increase in word forms per lemma creates considerable data sparsity. Fortunately, for many languages there exist large morphological lexi- cons, or better yet, morphological tools that can analyze any …

Content truncated to 500 words