Knowledge-Rich Morphological Priors for Bayesian Language Models

Proceedings of NAACL-HLT 2013, pages 1206–1215, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Knowledge-Rich Morphological Priors for Bayesian Language Models Victor Chahuneau Noah A. Smith Language Technologies InstituteChris Dyer Carnegie Mellon University Pittsburgh, PA 15213, USA {vchahune,nasmith,cdyer}@cs.cmu.edu Abstract We present a morphology-aware nonparamet- ric Bayesian model of language whose prior distribution uses manually constructed finite- state transducers to capture the word forma- tion processes of particular languages. This relaxes the word independence assumption and enables sharing of statistical strength across, for example, stems or inflectional paradigms in different contexts. Our model can be used in virtually any scenario where multinomial distributions over words would be used. We obtain state-of-the-art results in language modeling, word alignment, and un- supervised morphological disambiguation for a variety of morphologically rich languages. 1 Introduction Despite morphological phenomena’s salience in most human languages, many NLP systems treat fully inflected forms as the atomic units of language. By assuming independence of lexical stems’ vari- ous surface forms, this avoidance approach exacer- bates the problem of data sparseness. If it is em- ployed at all, morphological analysis of text tends to be treated as a preprocessing step to other NLP modules. While this latter disambiguation approach helps address data sparsity concerns, it has substan- tial drawbacks: it requires supervised learning from expert-annotated corpora, and determining the op- timal morphological granularity is labor-intensive (Habash and Sadat, 2006). Neither approach fully exploits the finite-state transducer (FST) technology that has been so suc- cessful for modeling the mapping between surface forms and their morphological analyses (Karttunen and Beesley, 2005), and the mature collections of high quality transducers that already exist for many languages (e.g., Turkish, Russian, Arabic). Much linguistic knowledge is encoded in such FSTs. In this paper, we develop morphology-aware non- parametric Bayesian language models that bring to- gether hand-written FSTs with statistical modeling and require no token-level annotation. The sparsity issue discussed above is addressed by hierarchical priors that share statistical strength across different inflections of the same stem by backing off to word formation models that piece together morphemes us- ing FSTs. Furthermore, because of the nonparamet- ric formulation of our models, the regular morpho- logical patterns found in the long tail of word types will rely more heavily on deeper analysis, while fre- quent and idiosyncratically behaved forms are mod- eled opaquely. Our prior can be used in virtually any generative model of language as a replacement for multino- mial distributions over words, bringing morphologi- cal awareness to numerous applications. For various morphologically rich languages, we show that: • our model can provide rudimentary unsuper- vised disambiguation for a highly ambiguous analyzer; • integrating morphology into n-gram language models allows better generalization to unseen words and can improve the performance of ap- plications that are truly open vocabulary; and • bilingual word alignment models also bene- fit greatly from sharing translation information 1206 across stems. We are particularly interested in low-resource sce- narios, where one has to make the most of the small quantity …

Content truncated to 500 words