Optimizing for Sentence-Level BLEU+1 Yields Short Translations

Proceedings of COLING 2012: Technical Papers, pages 1979–1994, COLING 2012, Mumbai, December 2012. Optimizing for Sentence-Level BLEU+1 Yields Short Translations Preslav NAKOV F rancisco GUZM ´AN Stephan VOGEL Qatar Computing Research Institute, Qatar Foundation Tornado Tower, floor 10, P.O. Box 5825, Doha, Qatar ④♣♥❛❦♦✈✱❢❤❡rr❡r❛✱s✈♦❣❡❧⑥❅q❢✳♦r❣✳q❛ ABSTRACT We study a problem with pairwise ranking optimization (PRO): that it tends to yield too short translations. We find that this is partially due to the inadequate smoothing in PRO’s BLEU+1, which boosts the precision component of BLEU but leaves the brevity penalty unchanged, thus destroying the balance between the two, compared to BLEU. It is also partially due to PRO optimizing for a sentence-level score without a global view on the overall length, which introducing a bias towards short translations; we show that letting PRO optimize a corpus-level BLEU yields a perfect length. Finally, we find some residual bias due to the interaction of PRO with BLEU+1: such a bias does not exist for a version of MIRA with sentence-level BLEU+1. We propose several ways to fix the length problem of PRO, including smoothing the brevity penalty, scaling the effective reference length, grounding the precision component, and unclipping the brevity penalty, which yield sizable improvements in test BLEU on two Arabic-English datasets: IWSLT (+0.65) and NIST (+0.37). KEYWORDS: Statistical machine translation, parameter optimization, MERT, PRO, MIRA. 1979 1 Introduction Early work on statistical machine translation (SMT) has relied on generative training using maximum likelihood parameter estimation. This was inspired by the noisy channel model (Brown et al., 1993), which asked for calculating the product of two components, a language model and a translation model, giving them equal weights. As mainstream research has moved towards combining multiple scores, the field has switched to discriminative tuning in a log-linear fashion. The standard approach has been to maximize BLEU (Papineni et al., 2002) on a tuning dataset using a coordinate descent optimization algorithm known as minimum error rate training (MERT), as proposed by Och (2003). MERT has dominated the SMT field for years, until the number of parameters in the log- linear model has gradually increased, in some cases to hundreds and even to hundreds of thousands of scores, which has called for new tuning algorithms since MERT was unable to scale beyond just a handful of parameters. Many alternatives to MERT have been proposed over the years, but it is only recently that some of them have gained popularity in the community, most notably, the margin infused relaxed algorithm (MIRA) (Chiang et al., 2008) and pairwise ranking optimization (PRO) (Hopkins and May, 2011). While the number of parameters that an optimizer can handle has become a major concern recently, there are many other important aspects that researchers have paid attention to, e.g., the performance of parameters when translating unseen test data, the speed of convergence, the stability across multiple reruns, the objective function being optimized (e.g., BLEU vs. an approximation of BLEU), the mode of learning (e.g., online vs. batch). Here we study a different, and …

Content truncated to 500 words