APRO: All-Pairs Ranking Optimization for MT Tuning

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1018–1023, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics APRO: All-Pairs Ranking Optimization for MT Tuning Markus Dreyer∗ SDL Research 6060 Center Drive Suite 150 Los Angeles, CA 90045 markus.dreyer@gmail.com Yuanzhe Dong SDL Research 6060 Center Drive Suite 150 Los Angeles, CA 90045 ydong@sdl.com Abstract We present APRO, a new method for machine translation tuning that can handle large feature sets. As opposed to other popular methods (e.g., MERT, MIRA, PRO), which involve ran- domness and require multiple runs to obtain a reliable result, APRO gives the same result on any run, given initial feature weights. APRO follows the pairwise ranking approach of PRO (Hopkins and May, 2011), but instead of rank- ing a small sampled subset of pairs from the k- best list, APRO efficiently ranks all pairs. By obviating the need for manually determined sampling settings, we obtain more reliable re- sults. APRO converges more quickly than PRO and gives similar or better translation results. 1 Introduction Machine translation tuning seeks to find feature weights that maximize translation quality. Recent efforts have focused on methods that scale to large numbers of features (Cherry and Foster, 2012), and among these, PRO has gained popularity (Pairwise Ranking Optimization, Hopkins and May (2011)). PRO’s goal is to find feature weights such that the resulting k-best list entries are ranked in the same way that an evaluation function (e.g., BLEU, Pap- ineni et al. (2002)) ranks them. To do this, it labels pairs of translations for each sentence as positive or negative, depending on the gold ranking of the two pair elements given by BLEU. A binary classifier is trained on these labeled examples, resulting in new feature weights, and the procedure is iterated. This ∗Markus Dreyer is now at Amazon, Inc., Seattle, WA. procedure would ordinarily be too expensive since there are O(k2) pairs per sentence, where both k and the number of sentences can be in the thousands, so billions of training examples would be produced per iteration. Therefore, Hopkins and May (2011) use subsampling to consider a small percentage of all pairs per sentence. We present APRO (All-Pairs Ranking Optimiza- tion), a tuning approach that, like PRO, uses pair- wise ranking for tuning. Unlike PRO, it is not lim- ited to optimizing a small percentage of pairs per sentence. Based on an efficient ranking SVM for- mulation (Airola et al. (2011), Lee and Lin (2014)), we find, in each iteration, feature weights that min- imize ranking errors for all pairs of translations per sentence. This tuning method inherits all the ad- vantages of PRO—it is scalable, effective, easy to implement—and removes its limitations. It does not require meta-tuning of sampling parameters since no sampling is used; it does not need to be run multi- ple times to obtain reliable results, like MERT (Och, 2003), PRO, MIRA (Chiang et al., 2008) and others, since it uses global optimization and …

Content truncated to 500 words