Batch Tuning Strategies for Statistical Machine Translation

Batch Tuning Strategies for Statistical Machine Translation Colin Cherry and George Foster National Research Council Canada {Colin.Cherry,George.Foster}@nrc-cnrc.gc.ca Abstract There has been a proliferation of recent work on SMT tuning algorithms capable of han- dling larger feature sets than the traditional MERT approach. We analyze a number of these algorithms in terms of their sentence- level loss functions, which motivates several new approaches, including a Structured SVM. We perform empirical comparisons of eight different tuning strategies, including MERT, in a variety of settings. Among other results, we find that a simple and efficient batch ver- sion of MIRA performs at least as well as training online, and consistently outperforms other options. 1 Introduction The availability of linear models and discriminative tuning algorithms has been a huge boon to statis- tical machine translation (SMT), allowing the field to move beyond the constraints of generative noisy channels (Och and Ney, 2002). The ability to opti- mize these models according to an error metric has become a standard assumption in SMT, due to the wide-spread adoption of Minimum Error Rate Train- ing or MERT (Och, 2003). However, MERT has trouble scaling to more than 30 features, which has led to a surge in research on tuning schemes that can handle high-dimensional feature spaces. These methods fall into a number of broad cate- gories. Minimum risk approaches (Och, 2003; Smith and Eisner, 2006) have been quietly capable of han- dling many features for some time, but have yet to see widespread adoption. Online methods (Liang et al., 2006; Watanabe et al., 2007), are recognized to be effective, but require substantial implementa- tion efforts due to difficulties with parallelization. Pairwise ranking (Shen et al., 2004; Hopkins and May, 2011) recasts tuning as classification, and can be very easy to implement, as it fits nicely into the established MERT infrastructure. The MERT algorithm optimizes linear weights relative to a collection of k-best lists or lattices, which provide an approximation to the true search space. This optimization is wrapped in an outer loop that iterates between optimizing weights and re-decoding with those weights to enhance the ap- proximation. Our primary contribution is to empiri- cally compare eight tuning algorithms and variants, focusing on methods that work within MERT’s es- tablished outer loop. This is the first comparison to include all three categories of optimizer. Furthermore, we introduce three tuners that have not been previously tested. In particular, we test variants of Chiang et al.’s (2008) hope-fear MIRA that use k-best or lattice-approximated search spaces, producing a Batch MIRA that outperforms a popular mechanism for parallelizing online learn- ers. We also investigate the direct optimization of hinge loss on k-best lists, through the use of a Struc- tured SVM (Tsochantaridis et al., 2004). We review and organize the existing tuning literature, provid- ing sentence-level loss functions for minimum risk, online and pairwise training. Finally, since random- ization plays a different role in each tuner, we also suggest a new method for testing an optimizer’s sta- bility (Clark et al., …

Content truncated to 500 words