A Comparative Investigation of Morphological Language Modeling for the Languages of the European Union

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 386–395, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics A Comparative Investigation of Morphological Language Modeling for the Languages of the European Union Thomas M¨uller, Hinrich Sch¨utze and Helmut Schmid Institute for Natural Language Processing University of Stuttgart, Germany {muellets,schmid}@ims.uni-stuttgart.de Abstract We investigate a language model that com- bines morphological and shape features with a Kneser-Ney model and test it in a large crosslingual study of European languages. Even though the model is generic and we use the same architecture and features for all lan- guages, the model achieves reductions in per- plexity for all 21 languages represented in the Europarl corpus, ranging from 3% to 11%. We show that almost all of this perplexity reduc- tion can be achieved by identifying suffixes by frequency. 1 Introduction Language models are fundamental to many natural language processing applications. In the most com- mon approach, language models estimate the proba- bility of the next word based on one or more equiv- alence classes that the history of preceding words is a member of. The inherent productivity of natural language poses a problem in this regard because the history may be rare or unseen or have unusual prop- erties that make assignment to a predictive equiva- lence class difficult. In many languages, morphology is a key source of productivity that gives rise to rare and unseen histories. For example, even if a model can learn that words like “large”, “dangerous” and “serious” are likely to occur after the relatively frequent his- tory “potentially”, this knowledge cannot be trans- ferred to the rare history “hypothetically” without some generalization mechanism like morphological analysis. Our primary goal in this paper is not to de- velop optimized language models for individual lan- guages. Instead, we investigate whether a simple generic language model that uses shape and mor- phological features can be made to work well across a large number of languages. We find that this is the case: we achieve considerable perplexity reduc- tions for all 21 languages in the Europarl corpus. We see this as evidence that morphological language modeling should be considered as a standard part of any language model, even for languages like English that are often not viewed as a good application of morphological modeling due to their morphological simplicity. To understand which factors are important for good performance of the morphological compo- nent of a language model, we perform an exten- sive crosslingual analysis of our experimental re- sults. We look at three parameters of the morpho- logical model we propose: the frequency threshold θ that divides words subject to morphological clus- tering from those that are not; the number of suffixes used φ; and three different morphological segmen- tation algorithms. We also investigate the differen- tial effect of morphological language modeling on different word shapes: alphabetical words, punctua- tion, numbers and other shapes. Some prior work has used morphological models that require …

Content truncated to 500 words