Leave-One-Out Phrase Model Training for Large-Scale Deployment

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 460–467, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Leave-One-Out Phrase Model Training for Large-Scale Deployment Joern Wuebker Human Language Technology and Pattern Recognition Group RWTH Aachen University, Germany wuebker@cs.rwth-aachen.de Mei-Yuh Hwang, Chris Quirk Microsoft Corporation Redmond, WA, USA {mehwang,chrisq}@microsoft.com Abstract Training the phrase table by force-aligning (FA) the training data with the reference trans- lation has been shown to improve the phrasal translation quality while significantly reduc- ing the phrase table size on medium sized tasks. We apply this procedure to several large-scale tasks, with the primary goal of re- ducing model sizes without sacrificing transla- tion quality. To deal with the noise in the auto- matically crawled parallel training data, we in- troduce on-demand word deletions, insertions, and backoffs to achieve over 99% successful alignment rate. We also add heuristics to avoid any increase in OOV rates. We are able to re- duce already heavily pruned baseline phrase tables by more than 50% with little to no degradation in quality and occasionally slight improvement, without any increase in OOVs. We further introduce two global scaling fac- tors for re-estimation of the phrase table via posterior phrase alignment probabilities and a modified absolute discounting method that can be applied to fractional counts. Index Terms: phrasal machine translation, phrase training, phrase table pruning 1 Introduction Extracting phrases from large amounts of noisy word-aligned training data for statistical machine translation (SMT) generally has the disadvantage of producing many unnecessary phrases (Johnson et al., 2007). These can include poor quality phrases, composite phrases that are concatenations of shorter ones, or phrases that are assigned very low proba- bilities, so that they have no realistic chance when competing against higher scoring phrase pairs. The goal of this work is two-fold: (i) investigating forced alignment training as a phrase table pruning method for large-scale commercial SMT systems and (ii) proposing several extensions to the training proce- dure to deal with practical issues and stimulate fur- ther research. Generative phrase translation models have the in- herent problem of over-fitting to the training data (Koehn et al., 2003; DeNero et al., 2006). (Wue- bker et al., 2010) introduce a leave-one-out proce- dure which is shown to counteract over-fitting ef- fects. The authors report significant improvements on the German-English Europarl data with the ad- ditional benefit of a severely reduced phrase table size. This paper investigates its impact on a num- ber of commercial large-scale systems and presents several extensions. The first extension is to deal with the highly noisy training data, which is automatically crawled and sentence aligned. The noise and the baseline prun- ing of the phrase table lead to low success rates when aligning the source sentence with the target sentence. We introduce on-demand word deletions, insertions, and backoff phrases to increase the suc- cess rate so that we can cover essentially the en- tire training data. Secondly, phrase table pruning makes out-of-vocabulary (OOV) issues even more pronounced. To avoid an increased …

Content truncated to 500 words