LAMB: A Good Shepherd of Morphologically Rich Languages

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 742–752, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics LAMB: A Good Shepherd of Morphologically Rich Languages Sebastian Ebert and Thomas M¨uller and Hinrich Sch¨utze Center for Information and Language Processing LMU Munich, Germany ebert@cis.lmu.de Abstract This paper introduces STEM and LAMB, em- beddings trained for stems and lemmata in- stead of for surface forms. For morpholog- ically rich languages, they perform signif- icantly better than standard embeddings on word similarity and polarity evaluations. On a new WordNet-based evaluation, STEM and LAMB are up to 50% better than standard em- beddings. We show that both embeddings have high quality even for small dimension- ality and training corpora. 1 Introduction Despite their power and prevalence, embeddings, i.e., (low-dimensional) word representations in vec- tor space, have serious practical problems. First, large text corpora are necessary to train high-quality embeddings. Such corpora are not available for un- derresourced languages. Second, morphologically rich languages (MRLs) are a challenge for stan- dard embedding models because many inflectional forms are rare or absent even in a large corpus. For example, a Spanish verb has more than 50 forms, many of which are rarely used. This leads to miss- ing or low quality embeddings for such inflectional forms, even for otherwise frequent verbs, i.e., spar- sity is a problem. Therefore, we propose to com- pute normalized embeddings instead of embeddings for surface/inflectional forms (referred to as forms throughout the rest of the paper): STem EMbed- dings (STEM) for word stems and LemmA eMBed- dings (LAMB) for lemmata. Stemming is a heuristic approach to reducing form-related sparsity issues. Based on simple rules, forms are converted into their stem.1 However, often the forms of one word are converted into several dif- ferent stems. For example, present indicative forms of the German verb “brechen” (to break) are mapped to four different stems (“brech”, “brich”, “bricht”, “brecht”). A more principled solution is lemmatiza- tion. Lemmatization unites many individual forms, many of which are rare, in one equivalence class, represented by a single lemma. Stems and equiva- lence classes are more frequent than each individual form. As we will show, this successfully addresses the sparsity issue. Both methods can learn high-quality semantic representations for rare forms and thus are most ben- eficial for MRLs as we show below. Moreover, less training data is required to train lemma embeddings of the same quality as form embeddings. Alterna- tively, we can train lemma embeddings that have the same quality but fewer dimensions than form em- beddings, resulting in more efficient applications. If an application such as parsing requires in- flectional information, then stem and lemma em- beddings may not be a good choice since they do not contain such information. However, NLP applications such as similarity benchmarks (e.g., MEN (Bruni et al., 2014)) and (as we show below) polarity classification are semantic and are largely 1In this paper, we use the term “stem” not in its linguistic meaning, but …

Content truncated to 500 words