Direct Error Rate Minimization for Statistical Machine Translation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 468–479, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Direct Error Rate Minimization for Statistical Machine Translation Tagyoung Chung∗ University of Rochester Rochester, NY 14627, USA chung@cs.rochester.edu Michel Galley Microsoft Research Redmond, WA 98052, USA mgalley@microsoft.com Abstract Minimum error rate training is often the pre- ferred method for optimizing parameters of statistical machine translation systems. MERT minimizes error rate by using a surrogate rep- resentation of the search space, such as N- best lists or hypergraphs, which only offer an incomplete view of the search space. In our work, we instead minimize error rate di- rectly by integrating the decoder into the min- imizer. This approach yields two benefits. First, the function being optimized is the true error rate. Second, it lets us optimize param- eters of translations systems other than stan- dard linear model features, such as distortion limit. Since integrating the decoder into the minimizer is often too slow to be practical, we also exploit statistical significance tests to ac- celerate the search by quickly discarding un- promising models. Experiments with a phrase- based system show that our approach is scal- able, and that optimizing the parameters that MERT cannot handle brings improvements to translation results. 1 Introduction Minimum error rate training (Och, 2003) is a com- mon method for optimizing linear model parame- ters, which is an important part of building good ma- chine translation systems. MERT minimizes an arbi- trary loss function, usually an evaluation metric such as BLEU (Papineni et al., 2002) or TER (Snover et al., 2006) from a surrogate representation of the search space, such as the N-best candidate transla- tions of a development set. Much of the recent work ∗ This research was conducted during the author’s intern- ship at Microsoft Research. on minimum error rate training focused on improv- ing the method by Och (2003). Recent efforts ex- tended MERT to work on lattices (Macherey et al., 2008) and hypergraphs (Kumar et al., 2009). Ran- dom restarts and random walks (Moore and Quirk, 2008) are commonly used to combat the fact the search space is highly non-convex, often with mul- tiple minima. Several problems still remain with MERT, three of which are addressed by this work. First, the N- best error surface explored by MERT is generally not the same as the true error surface, which means that the error rate at an optimum1 of the N-best er- ror surface is not guaranteed to be any close to an optimum of the true error surface. Second, most SMT decoders make search errors, yet MERT ig- nores the fact that the error surface of an error-prone decoder differs from the one of an exact decoder (Chang and Collins, 2011). MERT calculates an en- velope from candidate translations and assumes all translations on the envelope are reachable by the de- coder, but these translations may become unreach- able due to search errors. Third, MERT is only used to tune linear model …

Content truncated to 500 words