Multi-Metric Optimization Using Ensemble Tuning

Proceedings of NAACL-HLT 2013, pages 947–957, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Multi-Metric Optimization Using Ensemble Tuning Baskaran Sankaran, Anoop Sarkar Simon Fraser University Burnaby BC. CANADA {baskaran,anoop}@cs.sfu.ca Kevin Duh Nara Institute of Science & Technology Ikoma, Nara. JAPAN kevinduh@is.naist.jp Abstract This paper examines tuning for statistical ma- chine translation (SMT) with respect to mul- tiple evaluation metrics. We propose several novel methods for tuning towards multiple ob- jectives, including some based on ensemble decoding methods. Pareto-optimality is a nat- ural way to think about multi-metric optimiza- tion (MMO) and our methods can effectively combine several Pareto-optimal solutions, ob- viating the need to choose one. Our best performing ensemble tuning method is a new algorithm for multi-metric optimization that searches for Pareto-optimal ensemble models. We study the effectiveness of our methods through experiments on multiple as well as single reference(s) datasets. Our experiments show simultaneous gains across several met- rics (BLEU, RIBES), without any significant reduction in other metrics. This contrasts the traditional tuning where gains are usually lim- ited to a single metric. Our human evaluation results confirm that in order to produce better MT output, optimizing multiple metrics is bet- ter than optimizing only one. 1 Introduction Tuning algorithms are used to find the weights for a statistical machine translation (MT) model by min- imizing error with respect to a single MT evalua- tion metric. The tuning process improves the per- formance of an SMT system as measured by this metric; with BLEU (Papineni et al., 2002) being the most popular choice. Minimum error-rate train- ing (MERT) (Och, 2003) was the first approach in MT to directly optimize an evaluation metric. Sev- eral alternatives now exist: MIRA (Watanabe et al., 2007; Chiang et al., 2008), PRO (Hopkins and May, 2011), linear regression (Bazrafshan et al., 2012) and ORO (Watanabe, 2012) among others. However these approaches optimize towards the best score as reported by a single evaluation met- ric. MT system developers typically use BLEU and ignore all the other metrics. This is done despite the fact that other metrics model wide-ranging as- pects of translation: from measuring the translation edit rate (TER) in matching a translation output to a human reference (Snover et al., 2006), to capturing lexical choices in translation as in METEOR (Lavie and Denkowski, 2009) to modelling semantic simi- larity through textual entailment (Pad´o et al., 2009) to RIBES, an evaluation metric that pays attention to long-distance reordering (Isozaki et al., 2010). While some of these metrics such as TER, ME- TEOR are gaining prominence, BLEU enjoys the status of being the de facto standard tuning metric as it is often claimed and sometimes observed that optimizing with BLEU produces better translations than other metrics (Callison-Burch et al., 2011). The gains obtained by the MT system tuned on a particular metric do not improve performance as measured under other metrics (Cer et al., 2010), sug- gesting that over-fitting to a specific metric might happen without improvements in translation quality. In this …

Content truncated to 500 words