Tuning as Ranking

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1352–1362, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Tuning as Ranking Mark Hopkins and Jonathan May SDL Language Weaver Los Angeles, CA 90045 {mhopkins,jmay}@sdl.com Abstract We offer a simple, effective, and scalable method for statistical machine translation pa- rameter tuning based on the pairwise approach to ranking (Herbrich et al., 1999). Unlike the popular MERT algorithm (Och, 2003), our pairwise ranking optimization (PRO) method is not limited to a handful of parameters and can easily handle systems with thousands of features. Moreover, unlike recent approaches built upon the MIRA algorithm of Crammer and Singer (2003) (Watanabe et al., 2007; Chi- ang et al., 2008b), PRO is easy to imple- ment. It uses off-the-shelf linear binary classi- fier software and can be built on top of an ex- isting MERT framework in a matter of hours. We establish PRO’s scalability and effective- ness by comparing it to MERT and MIRA and demonstrate parity on both phrase-based and syntax-based systems in a variety of language pairs, using large scale data scenarios. 1 Introduction The MERT algorithm (Och, 2003) is currently the most popular way to tune the parameters of a sta- tistical machine translation (MT) system. MERT is well-understood, easy to implement, and runs quickly, but can behave erratically and does not scale beyond a handful of features. This lack of scalability is a significant weakness, as it inhibits systems from using more than a couple dozen features to discrimi- nate between candidate translations and stymies fea- ture development innovation. Several researchers have attempted to address this weakness. Recently, Watanabe et al. (2007) and Chiang et al. (2008b) have developed tuning methods using the MIRA algorithm (Crammer and Singer, 2003) as a nucleus. The MIRA technique of Chiang et al. has been shown to perform well on large-scale tasks with hundreds or thousands of fea- tures (2009). However, the technique is complex and architecturally quite different from MERT. Tellingly, in the entire proceedings of ACL 2010 (Hajiˇc et al., 2010), only one paper describing a statistical MT system cited the use of MIRA for tuning (Chiang, 2010), while 15 used MERT.1 Here we propose a simpler approach to tuning that scales similarly to high-dimensional feature spaces. We cast tuning as a ranking problem (Chen et al., 2009), where the explicit goal is to learn to correctly rank candidate translations. Specifically, we follow the pairwise approach to ranking (Herbrich et al., 1999; Freund et al., 2003; Burges et al., 2005; Cao et al., 2007), in which the ranking problem is reduced to the binary classification task of deciding between candidate translation pairs. Of primary concern to us is the ease of adoption of our proposed technique. Because of this, we adhere as closely as possible to the established MERT ar- chitecture and use freely available machine learning software. The end result is a technique that scales and performs just as well as MIRA-based tuning, but which …

Content truncated to 500 words