Stabilizing Minimum Error Rate Training

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 242–249, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics Stabilizing Minimum Error Rate Training George Foster and Roland Kuhn National Research Council Canada first.last@nrc.gc.ca Abstract The most commonly used method for training feature weights in statistical ma- chine translation (SMT) systems is Och’s minimum error rate training (MERT) pro- cedure. A well-known problem with Och’s procedure is that it tends to be sensitive to small changes in the system, particu- larly when the number of features is large. In this paper, we quantify the stability of Och’s procedure by supplying different random seeds to a core component of the procedure (Powell’s algorithm). We show that for systems with many features, there is extensive variation in outcomes, both on the development data and on the test data. We analyze the causes of this variation and propose modifications to the MERT proce- dure that improve stability while helping performance on test data. 1 Introduction Most recent approaches in SMT, eg (Koehn et al., 2003; Chiang, 2005), use a log-linear model to combine probabilistic features. Minimum Error- Rate Training (MERT) aims to find the set of log- linear weights that yields the best translation per- formance on a development corpus according to some metric such as BLEU. This is an essen- tial step in SMT training that can significantly improve performance on a test corpus compared to setting weights by hand. MERT is a difficult problem, however, because calculating BLEU as a function of log-linear weights requires decoding, which is an expensive operation. Moreover, be- cause this function is not differentiable, efficient gradient-based optimization algorithms cannot be used. Och’s procedure is the most widely-used ver- sion of MERT for SMT (Och, 2003). To reduce computational cost, it relies on the key technique of optimizing weights over n-best lists of transla- tion hypotheses rather than over all possible hy- potheses. This allows the most probable hypoth- esis under a given set of weights—and the corre- sponding BLEU score—to be found by enumer- ating n-best entries rather than decoding. Some variant on Powell’s algorithm (Press et al., 2002) is typically used to maximize BLEU in this set- ting. The n-best lists are constructed by alternat- ing decoding and BLEU maximization operations: decoding adds new hypotheses to the current lists, then BLEU is maximized over the lists to find new best weights for the subsequent decoding step, etc. This process continues until no new hypotheses are found. Och’s procedure works well in practice, usually converging after 10–20 calls to the decoder, far fewer than would be required to maximize BLEU directly with a general-purpose optimization algo- rithm. However, it tends to be sensitive to small changes in the system, particularly for large fea- ture sets. This is a well-known problem with Och’s procedure (Och et al., 2004). It makes it difficult to assess the contribution of features, be- cause the measured gain in performance due to a new feature can …

Content truncated to 500 words