Unsupervised Morphology Induction Using Word Embeddings

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1627–1637, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Unsupervised Morphology Induction Using Word Embeddings Radu Soricut Google Inc. rsoricut@google.com Franz Och∗ Human Longevity Inc. och@humanlongevity.com Abstract We present a language agnostic, unsupervised method for inducing morphological transfor- mations between words. The method re- lies on certain regularities manifest in high- dimensional vector spaces. We show that this method is capable of discovering a wide range of morphological rules, which in turn are used to build morphological analyzers. We evaluate this method across six different languages and nine datasets, and show significant improve- ments across all languages. 1 Introduction Word representations obtained via neural net- works (Bengio et al., 2003; Socher et al., 2011a) or specialized models (Mikolov et al., 2013a) have been used to address various natural language pro- cessing tasks (Mnih et al., 2009; Huang et al., 2014; Bansal et al., 2014). These vector representations capture various syntactic and semantic properties of natural language (Mikolov et al., 2013b). In many instances, natural language uses a small set of concepts to render a much larger set of mean- ing variations via morphology. We show in this pa- per that morphological transformations can be cap- tured by exploiting regularities present in word- representations as the ones trained using the Skip- Gram model (Mikolov et al., 2013a). In contrast to previous approaches that com- bine morphology with vector-based word represen- tations (Luong et al., 2013; Botha and Blunsom, 2014), we do not rely on an external morpholog- ical analyzer, such as Morfessor (Creutz and La- ∗Work done at Google, now at Human Longevity Inc. gus, 2007). Instead, our method automatically in- duces morphological rules and transformations, rep- resented as vectors in the same embedding space. At the heart of our method is the SkipGram model described in (Mikolov et al., 2013a). We fur- ther exploit the observations made by Mikolov et al (2013b), and further studied by (Levy and Gold- berg, 2014; Pennington et al., 2014), regarding the regularities exhibited by such embedding spaces. These regularities have been shown to allow infer- ences of certain types (e.g., king is to man what queen is to woman). Such regularities also hold for certain morphological relations (e.g., car is to cars what dog is to dogs). In this paper, we show that one can exploit these regularities to model, in a princi- pled way, prefix- and suffix-based morphology. The main contributions of this paper are as follows: 1. provides a method by which morphological rules are learned in an unsupervised, language- agnostic fashion; 2. provides a mechanism for applying these rules to known words (e.g., boldly is analyzed as bold+ly, while only is not); 3. provides a mechanism for applying these rules to rare and unseen words; We show that this method improves state-of-the-art performance on a word-similarity rating task using standard datasets. We also quantify the impact of our morphology treatment …

Content truncated to 500 words