Using a maximum entropy model to build segmentation lattices for MT

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 406–414, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Using a maximum entropy model to build segmentation lattices for MT Chris Dyer Laboratory for Computational Linguistics and Information Processing Department of Linguistics University of Maryland College Park, MD 20742, USA redpony AT umd.edu Abstract Recent work has shown that translating seg- mentation lattices (lattices that encode alterna- tive ways of breaking the input to an MT sys- tem into words), rather than text in any partic- ular segmentation, improves translation qual- ity of languages whose orthography does not mark morpheme boundaries. However, much of this work has relied on multiple segmenters that perform differently on the same input to generate sufficiently diverse source segmen- tation lattices. In this work, we describe a maximum entropy model of compound word splitting that relies on a few general features that can be used to generate segmentation lat- tices for most languages with productive com- pounding. Using a model optimized for Ger- man translation, we present results showing significant improvements in translation qual- ity in German-English, Hungarian-English, and Turkish-English translation over state-of- the-art baselines. 1 Introduction Compound words pose significant challenges to the lexicalized models that are currently common in sta- tistical machine translation. This problem has been widely acknowledged, and the conventional solu- tion, which has been shown to work well for many language pairs, is to segment compounds into their constituent morphemes using either morphological analyzers or empirical methods and then to trans- late from or to this segmented variant (Koehn et al., 2008; Dyer et al., 2008; Yang and Kirchhoff, 2006). But into what units should a compound word be segmented? Taken as a stand-alone task, the goal of a compound splitter is to produce a segmentation for some input that matches the linguistic intuitions of a native speaker of the language. However, there are often advantages to using elements larger than sin- gle morphemes as the minimal lexical unit for MT, since they may correspond more closely to the units of translation. Unfortunately, determining the op- timal segmentation is challenging, typically requir- ing extensive experimentation (Koehn and Knight, 2003; Habash and Sadat, 2006; Chang et al., 2008). Recent work has shown that by combining a vari- ety of segmentations of the input into a segmentation lattice and effectively marginalizing over many dif- ferent segmentations, translations superior to those resulting from any single single segmentation of the input can be obtained (Xu et al., 2005; Dyer et al., 2008; DeNeefe et al., 2008). Unfortunately, this ap- proach is difficult to utilize because it requires mul- tiple segmenters that behave differently on the same input. In this paper, we describe a maximum entropy word segmentation model that is trained to assign high probability to possibly several segmentations of an input word. This model enables generation of di- verse, accurate segmentation lattices from a single model that are appropriate for use in decoders that accept word …

Content truncated to 500 words